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Q a link to a URL or some other kind of link. The interpolation can be a linear interpolation or a spline interpolation technique. The 
^ video image can be communicated to a viewing system along with annotation corresponding to some of the video frames but not 
^ others of the video frames. The receiving system can determine the active areas for die others of said frames by interpolation. 
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5 MARKING OF MOVING OBJECTS IN VIDEO STREAMS 

6 

7 Leon A. Shirman 

8 Yakov Kamen 

9 Background of the Invention 

10 This application pertains to marking of objects in a video stream. 

11 It is known in the art to annotate or "mark" objects in a video stream with an 

12 interactive link. For example, suppose one were watching a video program of an 

13 automobile race, and there were a number of racing cars shown driving around a 

14 track. Also suppose that these cars were marked with interactive links. What this 

1 5 means is that one can position a cursor over the image of one of the cars and click on 



16 the image. This will cause the system controlling the video screen to display 

17 information that is linked to that image. This link can be in the form of a URL, and 

1 8 the system would display a web page depicting information concerning the car or 

19 driver that has been clicked on. 



20 When one "marks" an object, one establishes an area of the video screen 

21 surrounding that object (or near that object) as an "active area". If a viewer 

22 subsequently clicks on the active area, a link will be invoked. Referring back to the 

23 above example, if one of the racing cars is marked, there will be an active area 

24 surrounding or near the image of that racing car. If a viewer moves a cursor into that 

25 "active area" and clicks on that active area, the above-mentioned link will be invoked. 

26 In a video of a racing car, the car constantly moves, and its position on the 

27 video screen constantly changes. Accordingly, the active area corresponding to the 

28 car must constantly change, and "keep up" with the car. Thus, each frame of the 
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1 video must be marked with an appropriate active area for that car. Often, this is 

2 accomplished by image recognition. In other words, during an editing process, video 

3 editing apparatus recognizes if the car has moved from the right side of the video 

4 screen to the left side of the video screen. If this has happened, the editing apparatus 

5 will establish a new active area for that car on the left side of the screen for those 

6 video frames where the car is on the left side of the screen. Unfortunately, there are 

7 several problems with image recognition. For example, image recognition systems 

8 typically use the color of an object to track its location. If the object moves into an 

9 area where the lighting changes, e.g. because a shadow is cast on the object, that can 

10 prevent accurate image recognition. In addition, image recognition has other 

1 1 problems that can introduce errors into the process of marking an object. 

12 Image recognition is well known in the art and is discussed by J.P. Lewis in 

1 3 "Fast Normalized Cross-Correlation", 

14 http://www.idiom.com/'-'zilla/Papers/nvisionInterface/nip.htm; U.S. Patent 5,267,332, 

15 issued to Walch et al.; U.S. Patent 4,61 1,347, issued to Netravali et al.; U.S. Patent 

16 4,868,871, issued to Watson, III; R.O. Duda et al., "Pattern Classification and Scene 

17 Analysis", published by John Wiley and Sons in 1973; and R.C. Gonzalez et al., 

18 "Digital Image Processing", 3d edition, published by Addison- Wesley in 1992. 

19 Lewis, Walch, Netravali, Watson, Duda and Gonzalez are incorporated herein by 

20 reference. 

2 1 Instead of using image recognition apparatus, a person could mark each frame 

22 of the video manually. However, marking each frame of the video would be very 

23 tedious and time-consuming. What is needed is a novel method for marking moving 

24 objects that reduces the amount of human interaction required. 

2 
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I 

2 Summary 

3 A method in accordance with the invention comprises the step of annotating a 

4 video clip with a Hnk. (The video clip can be animation^ and as used herein, "video" 

5 includes animation.) This link can be a link to a web page or other source of 

6 information, or a link to application software. In one embodiment, a first set of 

7 frames within a video clip is annotated, e.g. using video editing apparatus. During 

8 this step of annotation, a portion of each annotated frame is designated as an "active 

9 area." This means that if one moves a cursor to the active area and clicks within the 
10 active area, a link associated with the active area will be executed. 

J J The active area is typically associated with an image of an object, e.g. a car 

12 racing around a track, an actor in a scene, a product displayed in a commercial, a 

1 3 cartoon character, or some other person or thing. Accordingly, during annotation of a 

14 frame, a person annotating the video clip will establish the location of the active area 

15 associated with that object. (The active area is typically a rectangular or circular 

16 region surrounding the object in question.) 

17 It would be extremely unwieldy to annotate evei^' frame in a video clip in this 

18 manner. Accordingly, in accordance with the invention, only a subset of the frames 

19 (e.g. every tenth frame) is annotated. The location of the active area in the other 

20 frames is established by interpolation. The location can be established by a linear 

21 interpolation technique. Alternatively, the location can be established by an 

22 interpolation technique that generates a curve, such as a spline interpolation 

23 technique. 

24 In another embodiment of a method in accordance with the invention, a subset 

25 of frames, for example two frames such as the first and last frames of a sequence of 

3 
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1 frames, are manually annotated. A video editing system then calculates the position 

2 of the active area for each frame between these manually annotated frames of the 

3 sequence based on an interpolation algorithm. The system then determines the 

4 difference between the interpolated position of the active area and the position of the 

5 object with which the active area is associated for each of the frames. This 

6 calculation is based on image recognition. The image recognition can be carried out 

7 by image recognition software. Alternatively, image recognition can be performed by 

8 a combination of hardware and software. (Image recognition can also be performed 

9 by hardware. The partitioning between hardware and software can be done in 

10 different ways.) Thereafter, the system highlights those frames that the system has 

1 1 determined to exhibit the greatest error. In other words, the system highlights those 

12 frames in which the distance between the active area and the object associated with 

13 that active area is greatest. One skilled in the art can define a distance between an 

14 active area and the object based on maximum, minimum or average distance between 

15 selected area and points within the object. 

16 In one embodiment, the system flags the frame with the largest error. In 

17 another embodiment, the system flags the n frames with the n largest errors, where n 

18 is an integer. In another embodiment, the system flags all frames in which the error is 

1 9 over a certain threshold. 

20 Thereafter, the person annotating the video clip can observe the flagged 

21 frames and decide whether he or she desires to manually annotate those frames. The 

22 person annotating the video clip also has the option of recalculating the interpolation. 

23 After recalculating the interpolation, the user can see if other frames are flagged that 

24 might require manual annotation. 



4 

SUBSTITUTE SHEET (RULE 26) 



wo 01/18676 PCT/USOO/24263 

1 Flagging may be accomplished by highlighting the flagged frame on a display 

2 or by providing some other form of output indicator, 

3 In accordance with another aspect of the invention, a video signal is sent to 

4 one or more video viewing systems so that one or more viewers can watch the video 

5 signal. The video signal can be broadcast to many viewers. The location of one or 

6 more active areas associated with a video signal is also communicated to the one or 

7 more video viewing systems for some of the frames of the video signal, but not other 

8 frames of the video signal. The one or more video viewing systems determine the 

9 location of the one or more active areas for the other frames by interpolation. In this 

10 way, instead of having to provide data to the video viewing systems concerning the 

1 1 active area locations for each frame, such data is provided only for selected frames, 

12 and the video viewing systems calculate where the active areas for the other frames 

13 should be by interpolating, e.g. using either a linear interpolation technique or a spline 

14 interpolation technique. Because the source of the video signal need not provide 

1 5 active area information for each frame of the signal, the bandwidth requirements for 

16 communicating this information are reduced. 

1 7 In one embodiment, not only is the location of the active area communicated 

1 8 in this way, but the size of the active area is communicated in this way as well. 

19 In accordance with another aspect of the invention, a video signal comprising 

20 a set of firames is communicated along with the location of an active area for some of 

2 1 the frames but not others of the frames. The active area corresponds to an object 

22 depicted in those frames. The object can be a product, a person, a cartoon character, 

23 or anything else that can be depicted in a video. As mentioned above, the active area 

24 is not communicated for the others of the frames, although the object is depicted in 

25 the others of the frames. A receiving system calculates the location of the active area . 
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1 by interpolation based on the location of the active area for said some of the frames. 

2 In this way, the bandwidth requirements for communicating the location of the active 

3 areas is reduced. 
4 

5 Brief Description of the Drawings 

6 Fig. 1 illustrates a display screen of a video editor/player used to annotate a 

7 video clip in accordance with the present invention. 

8 Fig. 2 illustrates the position of an object on a video screen along the y 

9 direction vs. frame number for a sequence of frames in a video clip. 

10 Fig. 3 is a flow diagram illustrating a method in accordance with a first 

1 1 embodiment of our invention. 

12 Fig. 4 illustrates the data architecture for a video editor/player used to mark a 

13 video clip in accordance with the invention. 

14 Fig. 5 is a block diagram of a signal transmission system and a reception and 

1 5 display system in accordance with another embodiment of the invention. 

16 Fig. 6 illustrates the position of the active area along the y axis vs. the frame 

1 7 number of a video clip. 
18 

19 Detailed Description 

20 Fig. 1 shows a display screen 100 depicting a typical graphical user interface 

21 ("GUI") for marking editors. Such an interface can be used during a method in 

22 accordance with our invention. Screen 1 00 shows a sequence of so-called key frames 

23 1 0 1 a to 1 0 1 p in the editing area. As is known in the art, video images comprise a 

24 sequence of frames. For digital video, each frame corresponds to an array of pixels. 

25 Each pixel is a data value corresponding to the color, brightness and saturation of a 
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1 small area of the frame. In certain video compression or encryption schemes, instead 

2 of storing every pixel array of every frame, the entire pixel array is stored only for 

3 certain "key" frames. Data is stored for the other frames that merely represent the 

4 difference between the image in those other frames and the preceding key frame. An 

5 example of a compression technique that uses this scheme is MPEG-2 (motion 

6 pictures expert group standard 2). 

7 For the case of linearly digitized analog video that is not compressed or 

8 encrypted, every nth frame (e.g. every 1 0^*^ frame) can be chosen as a key frame 

9 (along with frames corresponding to scene changes). As mentioned above, screen 1 00 

1 0 depicts a sequence of key frames 1 0 1 a to 1 0 1 p. 

1 1 Arrows 1 04a to 1 04d are not depicted on screen 1 00. Rather, arrows 1 04a to 

12 104d are included in Fig, 1 merely to symbolize the chronological sequence of the 

13 frames. 

1 4 Screen 1 00 includes a region 1 06 that depicts one of key frames 1 0 1 a to 1 0 1 p. 

15 The image in region 106 is magnified compared to the depiction of key frames 101a 

16 to 101 p elsewhere on the screen, and represents the frame that is currently being 

17 edited. In the example of Fig. 1, key frame lOlg is being edited. This is indicated by 

1 8 the presence of an icon or mark 1 0 1 g' provided at frame 1 0 1 g. 

19 Typically, screen 1 00 also depicts navigation bars and toolbars for functions 

20 such as fast forward, fast backwards, new file, etc. The system driving display 1 00 

2 1 includes software drivers for supporting the navigation bars and toolbars. These 

22 navigation bars and toolbars are not shown in Fig. 1 for sake of clarity. 

23 Also shown on screen 100 is a menu box 108 which enables a user to select a 

24 shape of an active area about an object that a viewer will be able to click on. Box 108 

25 enables an editor to select among many different shapes for the active area, only some 
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1 of which are depicted in Fig. 1 . As mentioned above, the active area is the location on 

2 a screen where an object of interest (e.g. a car or a person) is displayed. By clicking 

3 within the active area associated vAlh that object or person, one effectively clicks on 

4 that object or person. 

5 (When a viewer ultimately watches the video clip, the active area may or may 

6 not be displayed. For example, the system on which the video is displayed may 

7 highlight the active area. Alternatively, other systems on which the video is displayed 

8 may not highlight the active area. In yet other systems, the viewer may have the 

9 option to decide whether or not to highlight the active area.) 

10 Referring back to Fig. 1, box 1 10 is a menu box that allows the editor to 

1 ] introduce the link into the image, which is typically a URL. However, other types of 

12 links can be provided, including specialized links for launching desired application 

13 software. 

14 As seen in Fig. 1 , region 1 06 depicts a person 1 12, whose head 1 13 has been 

1 5 marked by a square 1 1 4 and linked to a link L 1 . (The square shape has been selected 

16 in menu box 108, and link LI has been selected in menu box 110.) Square 114 

1 7 symbolizes the active area. During a scene in a video, objects (e.g. the head of a 

1 8 person) typically move in a limited, non-jerked manner. (Good camera control during 

1 9 filming of a movie, television show or other video typically involves minimizing any 

20 jerking of the object of interest on the screen. Also, a small object moving in a jerked 

2 1 fashion, even if closely followed, would be difficult for a viewer to target when 

22 viewing the annotated video.) 

23 Square 1 14 represents the active area of the video screen on which a viewer 

24 will click if he wants to access information linked to person 1 12. For example, if the 

25 video is a soap opera, and the viewer wants to obtain more information about the 
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1 character played by person 112, he will click on person 112, and in particular, he will 

2 position the cursor within the active area represented by square 1 14 

3 During the video, person 1 12 typically moves about the set. Accordingly, it is 

4 necessary to have active area 1 14 move about the screen. This is necessary because if 

5 person 112 walks from the left side of the video screen to the right side of the video 

6 screen, and the viewer wants to find out more about person 1 1 2, he is going to want to 

7 click on current location of person 1 12. 

8 In principal, one could annotate each frame of the video to determine where 

9 acdve area 1 14 should be positioned, but this would be an enormously tedious, time- 

10 consuming task. Accordingly, it would be desirable to simplify this task. In 

1 1 accordance with the present invention, some of the frames are annotated by placing 

12 active area 1 14 at an appropriate location. (This can be done manually or through 

13 other techniques.) The other frames are annotated by an interpolation method 

14 described below. 

15 Fig. 2 illustrates the position of head 1 1 3 in a video along the y direction for 

16 each frame in a series of frames from a start frame A to an end frame D. Point 203 

1 7 represents the starting position of head 1 1 3 at the start of a scene and point 204 

18 represents the ending position of head 1 1 3 at the end of the scene. Line 206 traces the 

19 , position of head 1 13 between points 203 and 204 as the scene progresses. (Head 1 13 

20 changes position, e.g. as person 1 1 2 moves about the set during the video.) 

- 1 Line 208 shows a direct linear interpolation, resulting by having a straight line 

22 between the two positions 203 and 204. In other words, rather than annotating each 

23 frame manually, the position of active area 1 14 is automatically inserted into each 

24 frame by performing linear interpolation between positions 203 and 204. As can be 

25 seen, this is not necessarily the most accurate way of annotating the frames. One can 
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1 improve the linear interpolation by manually annotating more frames, and linearly 

2 interpolating between each frame for which a manual annotation has been performed. 

3 Fig. 2 also shows intermediate points 216, 21 8, indicating the position of head 

4 1 13 in frames B and C, respectively. In one embodiment, the position of active area 

5 1 14 is manually inserted (e.g. visually) for a set of frames, e.g. frames A, B, C and D. 

6 The position of active area 1 14 is established by linear interpolation between each 

7 point for which the position of active area 1 14 has been manually annotated. Thus, in 

8 Fig. 2, lines 224, 226 and 228 are the resuh of linearly interpolating between points 

9 203 and 2 1 6, points 2 1 6 and 2 1 8, and points 2 1 8 and 204 respectively. 

10 A more accurate way of interpolating is to use an interpolation technique that 

1 1 results in the generation of a curve that represents the position of the object from 

12 frame to frame. An example of such a technique is spline interpolation. When 

13 performing spline interpolation, one typically only needs the end points and one or 

14 two intermediate points, and the spline interpolation technique will accurately track 

15 the object. The spline will smooth the motion and eliminate jitter often seen by image 

16 recognition. 

17 It will be appreciated that one must not only establish the position of an object 

18 in the y direction, but also the x direction. Thus, during a method in accordance with 

19 our invention, the position of the active area v^U be established by interpolation for 

20 both the y and x directions. In addition, the object associated with the active area can 

21 expand or contract during the course of a video scene, e.g. because it moves closer to 

22 or fiirther from the camera. Accordingly, in one embodiment, the size of active area 

23 114 is also manually armotated for several frames during a scene, and the size of the 

24 active area is thereafter interpolated in the same manner as the position of the object. 

10 
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1 Fig. 3 is a block diagram illustrating a method in accordance with a first 

2 embodiment of the invention. Referring to Fig. 3, the method begins by loading a 

3 block of video frames into a memory (step 301). The memory can be a magnetic 

4 memory (e.g. a magnetic disk, magnetic tape or magneto-optic disk). The memory 

5 can also be an optical disk or a semiconductor memory such as a DRAM, SRAM, 

6 EPROM, or other type of memory. The video information can be either in analog 

7 form or in digital form. 

8 Thereafter, one manually annotates a subset of those frames (step 302). 

9 During this step of manual annotation, a set of frames are typically displayed on 

10 portions 101a to 101 p of the screen 100, and one of those frames can be selected for 

1 1 annotation at a time. The selected frame is shown on portion 106 of screen 100. The 

12 location, size and shape of the active area are established within those frames that are 

13 being annotated, and a link associated with that active area is established. Manual 

14 annotation can be performed using an input device such as a mouse, touchpad, 

15 joystick, trackball, cursor control keys, or other input device of the video editing 

16 system. During this step, the annotation information is stored in a memory, e.g. as 

17 described below. 



18 For the frames that are not manually annotated, annotation is done by 

19 interpolation (step 303). This can be done by any of the following steps: 

20 a) linear interpolation of the position of the active area and linear 

2 1 interpolation of the size of the active area; 

22 b) linear interpolation of the position of the active area with no change in 

23 size of the active area; 

24 c) spline interpolation of the position of the active area and spline 

25 interpolation of the size of the active area; or 
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1 d) spline interpolation of the position of the active area vsdth no change in 

2 the size of the active area. 

3 In lieu of linear or spline interpolation, other interpolation techniques can be used. By 

4 way of example, interpolation can be: 

5 a) between the start and end frames in a video clip; or 

6 b) interpolation from each manually annotated frame to the next manually 

7 annotated frame. (Typically, the start and end frames of the clip are 

8 among the frames that are manually annotated.) 

9 Referring to block 304, it is possible that interpolation can produce an error in 

10 position of the active region. In other words, there will be a difference between the 

1 1 position of an object and the position of the active area associated with that object. 

12 After interpolation, a person annotating the video clip can view the location of the 

1 3 active areas for the various frames at portions 1 0 1 a to 1 0 1 p of screen 100. If one of 

14 the frames shows a large discrepancy between the position of object 113 and active 

15 area 114, the person editing the video clip can manually annotate that frame, and have 

16 the editing system re-compute (e.g. by interpolation) all of the active area positions 

1 7 for the frames that have were not manually annotated. 

18 After establishing the position of active area 1 14, the person annotating the 

19 video clip chooses the form of the links represented for users. Instead of using a box, 

20 circle, ellipse or other shape, the active area could by depicted as a logo (e.g. a GIF 

21 file), a piece of animation (e.g. an animated GIF file) or other video stream segment. 

22 (GIF is an acronym for graphics interchange format.) As mentioned above, 

23 optionally, the active area can be visually indicated when the annotated video clip is 

24 later viewed. 

12 
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1 Referring to block 306, either after or before interpolation, a link to a URL can 

2 be associated with each active area. Thus, when a viewer clicks on that active area, 

3 that URL will be invoked. As mentioned above, the link need not be to a URL. It can 

4 also be a link to application software, or a source of information stored at a particular 

5 address. 

6 Fig. 4 illustrates a data architecture for an embodiment of a video editor/player 

7 used to mark a video clip. Such a video editor/player comprises a data mass storage 

8 device 406 that contains numerous video clips. The data mass storage device can be a 

9 magnetic disk, a magneto-optic disk, an optical disk, a semiconductor memory, or 

10 other type of memory storage device. 

1 1 Only one video clip 420a is shown in Fig. 4 for clarity, along with one 

12 matching annotation file 421a, although more than one video clip and annotation file 

13 may be stored in memory device 406. There can be more than one annotation file per 

14 video clip. The annotation file contains such things as the spline (either as a formula 

15 or as a reference to a predefined set of spline types) and its associated parameters, the 

16 link itself, and the color and size of the marker of the active area. Element 450 is a 

1 7 table or database, containing all the associated labels for each video sequence of 

18 jframes, thus serving as a directory for the video presentation. There may be, of 

19 course, several databases of shows and annotation files, possibly organized in a 

20 hierarchy structure, and they may share some of the files 420 or 421 depending on the 

21 edits done. 

22 In one embodiment, during annotation, data is loaded into a memor>' 

23 corresponding to the frame number, the location of the active area, its shape, its size, 

24 and the URL or other link associated with that active area. That data is provided to 

25 the viewer along with the video signal, typically in a separate file. This information 

13 
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1 can be provided as a separate data stream to the viewer. The viewer's video display 

2 system tracks the frame number, and for each frame, determines whether a cursor is 

3 positioned within one of the active areas established for that frame. 

4 In another embodiment, information concerning the location of the active area, 

5 its shape, its size, and the link are communicated and embedded as part of the video 

6 signal. This information can be provided during one of the blanking intervals of the 

7 signal or during some other portion of the signal that is not typically used. 
8 

9 Embodiment In Which Interpolation Errors Are Minimized or Corrected 

10 As can be seen in Fig. 2, when using interpolation to establish an active area 

11 on a video screen, there can be errors between the position of the marked object and 

12 the position of the active area. In accordance with another embodiment of the 

13 invention, these errors can be reduced or minimized through the following technique. 

14 1 . A block of video frames is loaded into a memory (step 301 of Fig. 3). 

15 2. A first set of n frames is manually annotated, where n is greater than or 

16 equal to 2 (step 302 of Fig. 3). The n frames can be the first and last frames in a video 

17 sequence. 

18 3. Interpolation is performed to establish an estimated position of an 

19 active area for those frames within the video sequence that are not manually annotated 

20 (step 303 of Fig. 3). Interpolation can be by linear interpolation. Alternatively, 

21 interpolation can be by a technique that results in a curve, e.g. a spline technique. As 

22 mentioned above, several interpolation techniques can be used to generate lines or 

23 curves, and the person annotating the video clip can choose among them. 

24 4. For each frame that is not manually axmotated, the video system 

25 calculates the distance between the active area and the object that the active area is 
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1 associated with. (The position of the active area is established by interpolation. The 

2 position of the object is established by automated image recognition techniques, e.g. 

3 as discussed above. As indicated above, image recognition may or may not be 

4 accurate.) 

5 5. The system flags those frames that have the greatest distance between 

6 active area 1 14 and the object 113 that the active area is associated with. The system 

7 can highlight those frames (e.g. one or more frames 101a to 101 p) that exhibit these 

8 distances and can place an icon or other marker at those frames, or it can use some 

9 other technique for identifying these frames to the user. The user can look at these 

10 frames, and verify whether the active area is, in fact, an unacceptable distance from 

1 1 the object with which it is associated. 

12 6. If the user believes that this distance is unacceptable, he or she can 

13 manually annotate the frames exhibiting the unacceptable error (step 304 of Fig. 3). 

14 7. After manually annotating the frames exhibiting the imacceptable 

1 5 error, the user can cause the system to recompute (e.g. by again interpolating) the 

16 position of the active area based on data including the new manual annotation. 

17 8. Optionally, the user can have the system take the new interpolated 

1 8 active area positions and establish another list of frames having the largest errors. If 

1 9 the user finds these errors to be acceptable, or merely artifacts relating to image 

20 recognition errors, he or she can ignore them. 

2 1 9. One then associates an active area shape (and optionally a visual 

22 indicator) and a link with the active area. 

23 As mentioned above, during step 304 of Fig. 3, one can cause the system to 

24 compute the distance between the active area and the object associated with that 

25 active area, and flag those frames shown at portions 101a to 10 Ip of screen 100 
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1 having unacceptable errors. The user can manually annotate those frames exhibiting 

2 unacceptable errors as described above. (This process can be used to adjust both the 

3 position (along both the x and y axes) and size of the active areas.) 
4 

5 Communicating Information Regarding the Active Areas to a Television Viewer 

6 After the video frames are marked with information concerning the active 

7 areas, it is necessary to communicate that information to the viewer's video system. 

8 Fig. 5 illustrates a video transmission system 500 comprising a data mass storage 

9 device 506 including a portion 520 that stores a video program to be transmitted. The 

10 video program can be any type of program — sporting events, soap operas, educational 

1 1 programs, cartoons and other animation, etc. Also included within mass storage 

12 device 506 is a portion 521 that contains annotation information. This annotation 

13 information includes a) the location of several active areas associated with various 

14 frames of the video program; b) the size of those active areas; and c) the link that 

15 those active areas are associated with. (The link can be in the form of a URL.) In 

16 other words, the annotation information might indicate that in a particular frame of the 

17 video program, there is a first active area located in the upper left portion of the video 

1 8 frame, and that active area has a certain size. If the viewer clicks on that active area, 

19 that should activate a link to a particular web page. 

20 Mass storage device 506 can be any type of mass storage device, e.g. a 

21 magnetic disk, magnetic tape, optical or magneto-optic media, or other data storage 

22 structure. 

23 Information from mass storage device 506 is provided to a data separation 

24 circuit 505 that separates out the video signal and provides that video signal to a 

25 conventional transmitter 504. (Data separation circuit 505 is typically a server.) A 
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1 television system 501 includes a receiver 511 for receiving the video signal. While 

2 Fig. 5 includes symbols indicating that the video signal is transmitted and received as 

3 conventional television radio signals, the video signals can be communicated by other 

4 means, e.g. an electrical cable, optical cable, microwave satellite communication, 

5 telephone lines, the internet, or other signal communication technique. 

6 The annotation information (e.g. the location and size of the active area, along 

7 with the URL information associated with the active area) is communicated to 

8 televisions system 501 via a communication medium 503, which can be the same as, 

9 or different from, the medium by which the video signal is communicated. Thus, 

10 communication medium 503 can be conventional radio signals, an electrical cable, 

1 1 optical cable, microwave, satellite communication, telephone line, the intemet, or 

12 other signal communication technique. 

13 The annotation information is received by a preprocessing unit 512. 

14 Preprocessing unit 512 calculates, via an interpolation technique, the location of each 

15 active area of each frame for those frames in which that data was not provided by 

16 transmission system 500. In one embodiment, the interpolation technique is a linear 

17 interpolation technique, e.g. as described above with respect to which Fig. 2. For 

18 example, suppose transmission system 500 provides the location for an active area 

1 9 every tenth frame. Fig. 6 illustrates that the y coordinate values for the active area for 

20 frames 0, 10, 20 and 30, indicated by points 600, 601, 602 and 603, respectively. If a 

2 1 linear interpolation technique is used, it will be assumed that the y coordinate for the 

22 active area v^ll be as established by lines 604, 605 and 606. Other interpolation 

23 techniques can be used, e.g. techniques that generate a curve instead of a straight line 

24 between specific points. For example, a spline interpolation technique can be used. If 

25 a spline interpolation technique is used, depending upon the spline calculation 
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1 parameters, it will be assumed that the y coordinate for the active area will be along * 

2 curve 607. 

3 Fig. 6 illustrates the y coordinate of the active area vs. frame number. Similar 

4 calculations are performed for the x coordinate of the active area, and the size of the 

5 active area. (The size of the active area can increase or decrease, e.g. depending on 

6 whether the object associated with the active area moves toward or away from the 

7 camera.) 

8 Referring again to Fig. 5, it is seen that the preprocessing unit 512 provides the 

9 video signal to a video display unit 514. Preprocessing unit 512 is coupled to a user 

10 interface. The user interface 513 permits a user to control certain features of the 

1 1 image displayed on display unit 514, e.g. whether a marker will be shown on the 

12 screen indicating the location of the active area. User interface 5 1 3 also permits a 

13 user to click on the various active areas to activate the links associated therewith. 

14 User interface 5 1 3 typically has an appropriate control mechanism such as a 

15 trackball, mouse, touchpad or joystick. This interface is used to position a visible 

16 cursor on video display unit 5 14 so that the viewer knows what object he is clicking 

17 on. Preprocessing unit 512 determines whether the cursor has been positioned over an 

1 8 active area. In one embodiment, preprocessing unit 512 comprises a microprocessor 

19 system. 

20 As mentioned above, in one embodiment of the invention, the active areas for 

21 selected frames are established, e.g. using a manual annotation system. The active 

22 areas are associated with objects depicted in the video frames. During this method, 

23 the location, size and shape of each active area is established for some (but not all) of 

24 the frames of the video clip. For other frames of the video clip, the location and size 

25 of the active areas is established by interpolating between the frames. In one 
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1 embodiment, the location and size of the active areas for those frames that were 

2 manually annotated are communicated to video system 501 . In other embodiments, 

3 the frames for which active locations and sizes are communicated are selected on 

4 other bases. For example, the location and size of the active areas can be 

5 communicated to receiving video system 501 for every tenth frame. Alternatively, the 

6 location and size of the active areas can be communicated only for the so-called "key 

7 frames" of the video clip. (As mentioned above, certain video compression 

8 algorithms communicate the entire pixel array only for certain key frames. For other 

9 frames, only information corresponding to the change between one frame and the next 

10 is commimicated. The key frames are those frames for which the entire pixel array is 

1 1 communicated.) 

12 Referring again to Fig. 5, in one embodiment, a workstation 550 can be 

13 coupled to video transmission system 500. Workstation 550 comprises a personal 

14 computer ("PC") 55 1 and a local memory storage device 552 (e.g. a disk) or other 

15 non-local storage (e.g. a server) that contains editing software 553. Workstation 550 

16 is used to annotate the video clip. In this example, workstation 550 is coupled via a 

17 communication line 554 to server 505, but many other ways to network workstation 

18 550 to server 505 can be used. In some embodiments, editor software 553 can run on 

19 server 505. In other embodiments, the files (both the video and annotation files) may 

20 reside on the local memory 552. 

2 1 While the invention has been described with respect to specific embodiments, 

22 those skilled in the art will appreciate that changes can be made in form and detail 

23 without departing from the spirit and scope of the invention. For example, the pixels 

24 of the video frames can be stored in terms of RGB values. (RGB refers to the values 

25 of the red, blue and green color components associated with the pixel.) Altematively, 
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1 the pixel values can be stored in terms of color, saturation and intensity. Screen 100 

2 and/or video display unit 5 1 4 can be a CRT screen, and LCD, a projection display, or 

3 other appropriate display technology. Different types of video editing/annotating 

4 systems can be used to practice the method of the present invention. The 

5 interpolation can be performed by a microprocessor or other computing device within 

6 the system. 

7 The method of the present invention can be used to provide or receive more 

8 than one active area in a video clip. 

9 Frames can be chosen for manual interpolation in different manners. For 

10 example, in one embodiment, every nth frame is manually interpolated, where n is an 

11 integer (e.g. 10). 

12 The apparatus chosen to perform the interpolation and editing can be a 

13 microprocessor-based system with a display screen executing video editing and/or 

14 image recognition software programs. 

15 In one embodiment, the first and last frames are not manually annotated. 

16 Accordingly, in such embodiments, the location and size of the active areas are 

17 extrapolated. 

1 8 When the video signal is sent to a viewer, both the video and annotation 

19 information can be sent as separate signals to receiving system 50 1 , or they can be 

20 communicated as parts of a combined signal. The combined signal can be provided 

21 by transmitter 504 via broadcast medium 502. Alternatively, the combined signal can 

22 be provided via any of the above-mentioned communications media, e.g. optical, 

23 cable, microwave transmission, satellite transmission, etc. 

24 The various aspects of the invention can be practiced independently of one 

25 another. For example, the annotation apparatus of Fig. 1 can be used without using 
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1 the signal communication apparatus of Fig. 5. Similarly, the communication 

2 apparatus of Fig. 5 can be used to communicate video clips annotated using methods 

3 other than the method described above with respect to Figs. 1 to 4. Accordingly, all 

4 such changes come within the present invention. 
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1 We claim: 

2 1. A method for establishing the location of an active area within a video 

3 clip, said video clip comprising a set of frames, said method comprising the acts of: 

4 (a) establishing the location of said active area for a first plurality of 

5 frames within said set of frames; 

6 (b) establishing the location of said active area for a second plurality of 

7 said frames within said set of frames by interpolation based on the location 

8 established during act (a) above. 
9 

2. Method of claim 1 wherein the locations established during the acts (a) 
II and(b) are stored in a memory device. 

12 

3. Method of claim 1 wherein act (a) comprises: 

1 4 displaying said first plurality of frames on a display device; and 

1 5 manually annotating said frames by storing within a memory information 

1 6 concerning the location of said active area within said first plurality of frames. 

17 

4. Method of claim 1 wherein said interpolation is linear interpolation. 

19 

2^ 5. Method of claim 1 wherein said interpolation is spline interpolation. 

21 

6. Method of claim 1 wherein said first plurality of frames are key 

23 frames. 
24 
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1 7. Method of claim 1 wherein said interpolation is performed by a 

2 computing device within an editing system. 

3 

4 8. Method of claim 1 further comprising the acts of: 

5 (c) displaying the active area for a third plurality of frames within said set 

6 of frames; 

7 (d) annotating at least some of said frames within said second plurality; 

8 and 

9 (e) establishing the position of said active area within said second plurality 
10 but not within said some of said frames by interpolation after performing act (d). 

11 

12 9. Apparatus comprising: 

13 a memory device storing a video clip, said video clip comprising a plurality of 

14 video frames; 

15 an input device for inputting information into said apparatus indicating the 

16 location of an active area within some of said video frames; and 

17 a computing device within said apparatus for establishing the location of said 

18 active area for other frames within said plurality. 
19 

20 10. Apparatus of claim 1 0 wherein said computing device establishes the 

2 1 location of said active area for said other frames within said plurality of interpolation. 
22 

23 11. A method comprising: 
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1 providing a video clip comprising a set of frames, a first plurality of frames 

2 within said set being annotated with information indicating the location of an active 

3 area, said active area being associated with an object shown within said video clip; 

4 establishing location data for said active area for those frames that are not 

5 within said first plurality by interpolation; 

6 using automated image recognition to identify one or more of said frames that 

7 are not within said first plurality for which the distance between said active area and 

8 said object is large compared to others of said frames that are not within said first 

9 plurality. 
10 

11 12. Method of claim 1 1 wherein said automated image recognition identifies 

12 frames in which the distance between said active area and said object exceeds a 

13 threshold value. 

14 

15 13. Method of claim 1 1 wherein said automated image recognition identifies the n 

16 frames for which the distance between the active area and object are largest, where n 

17 is an integer. 
18 

19 14. Method of claim 1 1 fiirther comprising displaying said one or more frames on 

20 a display so that a viewer can see the distance between said active area said object in 

21 said one or more frames. 

22 

23 1 5. Method of claim 1 1 fiirther comprising annotating at least one of said one or 

24 more frames after said act of using. 
25 
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1 16. Method of claim 1 5 further comprising establishing location data for said 

2 active area for those frames that are not within said first plurality and are not within 

3 said at least one frame by interpolation. 
4 

5 1 7. Method of claim 1 6 further comprising displaying the frames whose location 

6 data has been established by interpolation so that a user can see how far the active 

7 area is from the object. 
8 

9 1 8. Method of claim 1 1 wherein an operator annotates said first plurality of fi-ames 

1 0 with said information prior to said act of establishing. 
11 

12 19. Method of claim 1 8 wherein said annotation is performed by storing data in a 

13 memory indicating the position of said active area for said first plurality of fi-ames. 
14 

15 20. Method of claim 1 1 wherein said video clip is stored in a memory device. 
16 

17 21 Method of claim 20 wherein said information indicating the location of an 

1 8 active area is stored in the same memory device as said video clip. 
19 

20 22. Method of claim 2 1 wherein said video clip and said information indicating 

21 the location of an active area are located in different files within said memory device. 
22 

23 23 . Method of claim 2 1 wherein said information indicating the location of said 

24 active area is embedded within said video clip. 
25 
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1 24. Method comprising the steps of: 

2 providing a video clip, said video clip comprising a group of frames, said 

3 video clip depicting an object; 

4 annotating some of said frames within said group of frames, but not others of 

5 said frames within said group of frames, by storing in a memory die location of an 

6 active area within said some of said frames associated with said object; 

7 establishing the location of said active area for said others of said frames by 

8 interpolation; 

9 using automated image recognition to determine those frames within said 

10 others of said frames for which the distance between said active area and said object 

1 1 is large compared to others of said frames that are not within said first plurality. 
12 

13 25. Apparatus comprising: 

14 a memory structure for storing a video clip, said video clip comprising a set of 

15 frames, a first plurality of frames within said set being annotated with information 

1 6 indicating the location of an active area, said active area being associated with an 

1 7 object shown within said video clip; 

18 means for establishing location data for said active area for those frames that 

19 are not in said first plurality by interpolation; 

20 means for using automated image recognition to identify one or more of said 

2 1 frames for which the distance between said active area and said object is large 

22 compared to others of said frames that are not within said first plurality. 
23 

24 26. A method comprising the steps of: 

25 receiving a video image; 
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1 displaying said video image as a sequence of frames; 

2 receiving information corresponding to the location of at least one active area 

3 for some of said frames but not others of said frames; and 

4 determining the location of said active area for said others of said frames 

5 based on said received information. 
6 

7 27. Method of claim 26 wherein said step of determining is performed by linear 

8 interpolation. 
9 

10 28. Method of claim 26 wherein said step of determining is performed by spline 

1 1 interpolation. 
12 

13 29. Method of claim 26 wherein information corresponding to the size of said 

14 active area is received for said some of said frames but not said others of said frames, 

15 said method also comprising the step of determining the size of said active area for 

16 said others of said frames by interpolation. 
17 

18 30. Method of claim 26 wherein said active area is associated with a link, said 

1 9 method further comprising the step of activating said link. 
20 

21 31. Method of claim 26 wherein said link is a link to an internet page. 
22 

23 32. Method of claim 26 further comprising activating said link. 
24 
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1 33. Method of claim 32 wherein said displaying comprises displaying said video 

2 on a display screen and said activating comprises clicking on an active area on said 

3 display screen. 

4 

5 34, Method comprising: 

6 communicating a video clip comprising a set of video frames, some of the 

7 video frames within said set of video frames being annotated such that an active area 

8 is established for said some of said video frames; 

9 communicating the location of said active area for said some of the video 

10 frames within said set of video frames and not others of said video fi-ames within said 

1 1 set of video frames. 
12 

13 35. Method of claim 34 wherein said video clip is communicated through a first 

14 communication mediimi and said location of said active area is communicated 

15 through a second communication medium that is different from said fu:st 

1 6 communication medium. 
17 

18 36. Method of claim 34 further comprising: 

19 receiving said video clip and the location of said active area for some of said 

20 video frames within said set of video frames and not said others of said video frames; 

21 and 

22 establishing positions for said active area within said others of said video 

23 frames by interpolation. 
24 
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1 37. Method of claim 36 wherein said active area is associated with an object, said 

2 object being shown in each of said video frames within said set of video frames. 

3 

4 38. A system comprising; 

5 a receiver for receiving a video signal; 

6 a display coupled to said receiver for displaying a video image corresponding 

7 to said video signal, said video image comprising a set of frames; 

8 a circuit receiving information indicating the position of at least one active 

9 area for some of said frames and not others of said frames, said circuit determining 

10 the location of said active area for said others of said frames by interpolation based on 

1 1 the location of said active area for said some of said frames. 
12 

13 39. System of claim 38 further comprising a transmission system for transmitting 

14 said video signal and said information corresponding to the position of at least one 

1 5 active area for said one of said frames and not said others of said frames. 
16 

17 40. Apparatus comprising: 

18 a first memory storage device storing a video signal, said video signal 

19 comprising a set of frames; 

20 a memory storage device storing the location of an active area for some of said 

21 frames; 

22 a transmitter for transmitting said video signal and said location of said active 

23 area for said some of said frames but not others of said frames, whereby a system 

24 receiving said video signal and said location of said active area for said some of said 
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1 frames can establish a location for said active area for said others of said frames by 

2 interpolation. 

3 

4 41. Apparatus comprising: 

5 a memory storing a video clip; 

6 a structure for establishing the location of an active area for some of the 

7 frames of the video clip but not other frames of the video clip; and 

8 a transmission system for transmitting the video clip and the location of the 

9 active area for said some of said frames of the video clip but not other frames of the 

10 video clip. 
11 

12 42. Apparatus of claim 41 further comprising a receiving unit for receiving and 

13 displaying the video clip and establishing the location of said active area for said other 

14 frames by interpolation. 
15 

16 43. A method for establishing the location of an active area within a series of 

1 7 frames of images, said method comprising the acts of: 

1 8 (a) establishing the location and/or size of said active area for a first 

19 plurality of frames within said series of frames; 

20 (b) establishing the location and/or size of said active area for a second 

2 1 plurality of said frames within said series of frames by calculation based on the 

22 location established during act (a) above. 
23 

24 44. Apparatus of claim 40 wherein said memory storage device stores the location 

25 of said active area for some of said frames but not others of said frames. 
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METHODS, APPARATUS. AND SYSTEMS FOR STORING, 
RETRIEVING AND PLAYING MULTIMEDIA DATA 



5 BACKGROUND OF THE INVENTION 

Field of the Invention 

The invention relates generally to improvements in computer systems. 
More particularly, the invention relates to methods, apparatus, and systems 
storing multimedia content such as audio, text, image, and graphical content in a 

10 cache directory. 

Discussion of the Related Art 

Prior art graphics processing storage medium, sometimes called a cache 
system, is known to those skilled in the art. For example, a conventional 
caching system is typically composed of a small fast storage device that 

1 5 contains a "snapshot" of information originally received from a larger, slower 
source. The snapshot is considered by the particular implementation to be the 
most relevant information to the processing occurring during the current time 
period. 

In the context of Intemet content, a "cache" is a file, database, directory, 
20 or set of directories disposed in a computer file system. The cache stores 
content that has been previously retrieved generated or otherwise produced. 
Intemet browsers and editors use cache directories to store content. The cached 
content is used in place of remote content whenever possible in order to 
decrease retrieval latencies. Therefore, many web browsers and text editors 
25 save Intemet and other text and graphical content in a cache directory in order to 
reduce access times. This content is usually stored in its original form [for 
example, hypertext markup language (HTML) and accompanying images]. 

A problem with this technology has been that to view content based on 
the image data stored in the cache typically requires layout and rendering of the 
30 data. If the data upon which content is based does not change, the process of 
rendering need only occur once to a display buffer. When information is 
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changed, the information must be re-rendered to reflect the desired change. For 
complex graphics scenes re-rendering can require massive processing for only 
incremental changes in the scene or particular graphic. The layout and 
rendering processes are time consuming and require processor resources, 
5 Therefore, what is required is solution that provides play of multimedia content 
more efficiently in terms of time and processor resources. 

Heretofore, the requirements of timely and processor efficient play of 
multimedia content have not been fully met. What is needed is a solution that 
simultaneously addresses these requirements. The invention is directed to 
1 0 meeting these requirements, among others. 

SUMMARY OF THE INVENTION 

A primary goal of the invention is to provide timely and processor 
efficient display of multimedia content. In accordance with these goals, there is 
1 5 a particular need for a storage medium that includes multimedia content and the 
semantic content of the multimedia content. A storage medium including both 
the multimedia content and the semantic content is referred to herein as a 
rendered cache. 

For various embodiments of the invention, the semantic content can 
20 include locations, sizes, shapes, and target imiversal resource identifiers of 

hyperlinks, multimedia element timing, and other content play instructions. The 
very fast play of content stored in the rendered cache is due to the elimination of 
the steps of laying out the content, rendering the content, and generating the 
semantic representation of the content. These steps are required each time the 
25 content is played after retrieval from a conventional cache. The only steps 
required for playing content from the rendered cache are to read the rendered 
content, read the semantic content, restore the semantic representation, and play 
the content. 

A traditional web browser visiting a web site that resides in a rendered 
30 cache provides an almost instantaneous display of the web site. The caching 
mechanism provided by various embodiments of the invention is independent of 
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content file format and the stored semantic content file format. As long as a 
client application, such as a content browser, can recognize and play the 
multimedia content and recognize and interpret the semantic content, the 
application can realize the benefits provided by the rendered cache. Thus, it is 
5 possible to simultaneously satisfy the above-discussed requirements of timely 
and processor efficient display of multimedia content, which, in the case of the 
prior art, £u:e not simultaneously satisfied. 

A first aspect of the invention is provided as an embodiment that is 
based on a method, implemented in at least one computer, for storing 

10 multimedia data. The method for storing multimedia data comprises detecting 
multimedia content, generating a semantic representation of a rendered 
representation of the multimedia content firom the play instructions, storing the 
rendered representation in a storage medium, and storing data corresponding to 
the semantic representation in the storage medium. The multimedia content 

15 includes play instructions and at least one multimedia element. The at least one 
multimedia element includes at least one of graphical images, audio, text, and 
fiiU motion video. The play instructions include at least one of timing of the 
multimedia content and ordering of the multimedia content. The semantic 
representation describes at least one of characteristics of the rendered 

20 representation, and relationships between different multimedia elements 
disposed in the rendered representation. 

A second aspect of the invention is provided as an embodiment that is 
based on a method, implemented in at least one computer, for storing 
multimedia data. The method for storing multimedia data comprises detecting 

25 multimedia content including layout instructions, and laying out the multimedia 
content according to the layout instructions to form rendering instructions and a 
semantic representation of a rendered representation of the multimedia content. 
The method also includes rendering the multimedia content according to the 
rendering instructions to produce the rendered representation, storing the 

30 rendered representation in a storage medium, and storing data corresponding to 
the semantic representation in the storage medixmi. 
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A third aspect of the invention is provided as an embodiment that is 
based on a method, implemented in at least one computer, for retrieving 
multimedia data. The method for retrieving multimedia data comprises 
processing resources of a first computer of the at least one computer detecting a 
5 request for requested multimedia content, and processing resources coupled 
with the first computer determining whether data corresponding to the requested 
multimedia content is disposed in a storage medium. The storage medixmi is 
coupled with the first computer and includes rendered representations of 
muhimedia content and semantic content. Embodiments according to the third 
10 aspect of the invention also include responding to a determination that data 
corresponding to the requested multimedia content are disposed in the storage 
medium by retrieving a rendered representation of the requested multimedia 
content; and retrieving semantic content corresponding to the requested 
multimedia content, 

1 5 A fourth aspect of the invention is implemented in an embodiment that is 

based on a rendered cache comprising a storage medium, and an indexing 
mechanism adapted to store and retrieve a rendered representation of the 
multimedia content formatted for rapid play and semantic content of the 
multimedia content. 

20 A fifth aspect of the invention is implemented in an embodiment that is 

based on a client. The client comprises processing resources adapted to detect a 
rendered representation of multimedia content and semantic content of the 
rendered representations, and processing resources adapted to respond to 
detecting the rendered representation of the multimedia content and the 

25 semantic content by playing at least a portion of the rendered representation 
according to the semantic content. 

A sixth aspect of the invention is implemented in an embodiment that is 
based on a system for using multimedia content. The system comprises web 
crawler processing resources adapted to access the multimedia content ftx)m 

30 source data storage, rendering processing resources, and a rendered cache as 
described above as the fourth aspect of the invention. The rendering processing 
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resources are adapted to generate a semantic representation of a rendered 
representation of the multimedia content, and format the semantic 
representation as semantic content, and render the multimedia content into the 
rendered representation, the rendered representation is formatted for rapid play. 

5 A seventh aspect of the invention is implemented in an embodiment that 

is based on a system for accessing multimedia content. The system for 
accessing multimedia comprises a rendered cache as described above as the 
fourth aspect of the invention, and rendering processing resources adapted to 
convert the multimedia content into the rendered representation, the rendered 

10 representation is formatted for rapid play, and create a graphical representation 
of the multimedia content. 

An eighth aspect of the invention is implemented in a method for 
playing multimedia content. The method comprises retrieving a rendered 
representation of the multimedia content from a storage medium, and retrieving 

15 semantic content of the rendered representation from the storage medium. The 
method includes browser processing resources reading the rendered 
representation and the semantic content, and the browser processing resources 
restoring a semantic representation based on the semantic content. The method 
includes the browser processing resources transmitting an active portion of the 

20 rendered representation to a client, and transmitting an active portion of the 
semantic content corresponding to the active portion of the rendered 
representation to the client. The active portion of the rendered representation is 
one of a portion of the rendered representation presently being played, and a 
portion of the rendered representation to be played rapidly after transmitting. 

25 The mediod also includes client processing resources detecting the active 
portion of the rendered representation and the active portion of the semantic 
content, and the client processing resources playing the active portion of the 
rendered representation. 

30 BRIEF DESCRIPTION OF THE DRAWINGS 
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These, and other, goals and aspects of the invention will be better 
appreciated and understood when considered in conjunction with the following 
description and the accompanying drawings. Various embodiments of the 
invention are illustrated in the drawings accompanying and forming a part of 
5 this specification, wherein like reference characters (if they occur in more than 
one view) designate the same parts. It should be noted that the features 
illustrated in the drawings are not necessarily drawn to scale. 

FIG. 1 illustrates a schematic block diagram of a conventional method 
for retrieving and playing multimedia content, appropriately labeled "PRIOR 
10 ART". 

FIG. 2 illustrates a schematic block diagram of a process overview for 
retrieving and playing multimedia content using a rendered cache, representing 
an embodiment of the invention. 

FIG. 3 illustrates a schematic block diagram including render process 
1 5 details, representing an embodiment of the invention. 

FIG. 4 illustrates a schematic block diagram including play process 
details, representing an embodiment of the invention, 

FIGS. 5A-5B illustrate screen shots of portions of a Toronto Exchange 
Internet page, representing an embodiment of the invention. 
20 FIG. 6A illustrates the timing of play of different multimedia elements 

for an example of multimedia content that does not require layout, representing 
an embodiment of the invention. 

FIGS. 6B-6D illustrate different images included in the multimedia 
content not including layout example representing an embodiment of the 
25 invention. 

FIG. 7 illustrates a communications system including a rendered cache, 
representing an embodiment of the invention. 

DESCRIPTION OF PREFERRED EMBODIMENTS 

30 The invention and the various features and advantageous details thereof 

are explained more fully with reference to the non-limiting embodiments that 
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are illustrated in the accompanying drawings and detailed in the following 
description of preferred embodiments. Descriptions of well-known components 
and processing techniques are omitted so as not to unnecessarily obscure the 
invention in detail. It should be understood, however, that the following 
5 description, while indicating preferred embodiments of the invention and 
numerous specific details thereof, is given by way of illustration and not of 
limitation. Many changes and modifications may be made within the scope of 
the invention without departing from the spirit thereof, and the invention 
includes all such modifications. 
10 Methods, apparatus and systems are described for storing multimedia 

content using a process for caching fully rendered documents in a way that 
significantly increases content viewing speeds, navigation in a hyperlink 
document, while decreasing processing requirements. 
Definitions 

15 The following terms are used in the description of various embodiments 

of the invention provided herein. 

Content: Text and graphical information that require a layout and/or 
rendering process in order to be viewed on a computer, television or other 
display device. Other terms for content include web-page, document, Internet 
20 content, hypertext markup language (HTML), extensible Markup Language 
(XML), and Television Markup Language (TVML). Content can also include 
non-graphical information such as audio. 

Content Browser: A computer program designed to retrieve, display or 
navigate content. Examples include Internet web browsers, 
25 HTML/XML/Standard Generalized Markup Language (SGML) editors, word 
processors, and Intemet web proxies. 

HTML: The de facto Intemet content standard. HTML includes a set of 
markup rules that describe the layout of Intemet content. Browsers use this 
markup to layout and render the HTML for viewing computer monitors, 
30 televisions, or other displays. 

Markup: Notation used to describe the syntactic and semantic features 
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of a content document. 

Multimedia Content: Multimedia elements used for playing a 
presentation for a user. The multimedia elements can include graphical images 
(including rendered HTML), audio, text, and full motion video. 
5 Navigation: The process of selecting an indexing indication, such as a 

URI in the form of a hyperlink, from displayed content to access further content. 

Presentation: Content that references at least one multimedia element. 
Presentations include play instructions that can be used to define the timing, 
order, and position of the multimedia plays. The play instructions can include 
10 the size, shape and target of all hyperlinks, information on interactive elements 
(like HTML forms), and Meta values. 

Render: The process of generating a graphical representation of data 
that can be viewed on a display. For example, web browsers render HTML 
pages into graphical images that can be viewed on a computer monitor or 
15 television. Also the process of generating or converting multimedia data 
(images, audio, text, full motion video) into a format that can be played. 

Rendered Cache: Various embodiments of the invention use the 
concept of a rendered cache to mean a cache of content that is not only 
generated (or retrieved) from a multimedia content data source, such as the 
20 Internet; but also is rendered and ready for rapid play. The rendered cache can 
include two types of objects: multimedia content and semantic content. The 
multimedia content stored in the rendered cache is content that has been 
rendered and is ready for very quick display. Semantic content includes a 
description of the semantic features or representation of the rendered content. 
25 Examples of semantic features include the location, size, shape and target of 
hyperiinks, the timing, location, and size of animated graphics interchange 
format (GIF) frames, the size and relative location of HTML frames, 
information on HTML forms, HTML meta values, presentation play timing, and 
other play instructions. A more detailed description of the rendered cache is 
30 provided in the Process Description section below. 

Semantic Representation: A description of the characteristics. 
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attributes, logical structure, and features of multimedia elements (or objects) 
that form a rendered representation of multimedia content, or a portion thereof. 
The data can also describe the relationships between different multimedia 
elements within a particular presentation portion, and the way various elements 
5 of the multimedia content are accessed and manipulated. The semantic 

representation is typically generated during the layout process and is structured 
such that the semantic representation can be saved as formatted and indexed 
semantic content in a file or database, and rapidly restored from the semantic 
content. The semantic content can be stored along with the multimedia content 

10 or as one or more separate indexed files. The semantic representation is 

independent of the format of the stored semantic content. The Document Object 
Model (DOM) is one type of semantic representation and is adapted for use with 
HTML and XML docxmients. 

TVML: Some embodiments of the invention (including the 

15 VirtualModem™ presentation system provided by Interactive Channel, Inc. 
located in London, Ontario, Canada) use an XML language called television 
markup language (TVML) to describe multimedia content. TVML includes 
markup to describe how to play multimedia content. The multimedia content 
can include text (including HTML), graphical images, audio, text, and full- 

20 motion video. TVML can include markup to describe when each multimedia 
component should be played relative to the other multimedia components. 

URI: A Universal Resource Identifier (or URI) is an Intemet standard 
term for all types of names and addresses that refer to content. The term URI 
encompasses terms such as filename, hyperlink, and Universal Resource 

25 Locator (URL). 

VMML: An XML markup language (called VMML - VirtualModem™ 
Markup Language) used to store semantic representations of rendered 
multimedia content by various embodiments of the invention, such as the 
VirtualModem™ presentation system. 

30 XML: A markup language used to describe other markup languages, 

such as HTML and TVML. 
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Process Description 

Various embodiments of the invention include methods, implemented in 
at least one computer, for storing and retrieving multimedia data. These 
5 methods navigate and play multimedia content with increased speed and 
decreased computer processing by using different types of data objects to 
represent the multimedia data. A first data object type includes pre-rendered 
multimedia content data, A second data object type includes a semantic 
representation of the pre-rendered multimedia content. These data object types 

10 can be stored as separate files or can be contained in the same file. 

Prior art methods for retrieving and playing multimedia content are 
represented by Figure 1, which includes a traditional cache 1 10. After detecting 
a request to play multimedia content (at step 120), retrieving processing 
resources, such as those disposed in a web browser, retrieve the corresponding 

15 multimedia content data. A traditional web browser, such as Netscape 

Navigator, Netscape Communicator, or Microsoft® Internet Explorer, when 
coupled with a traditional cache 1 10, then performs the steps described below in 
response to each and every play request 120. 

After retrieving the content (e.g., the HTML content description), the 

20 content is read (step 130) from either a traditional cache 1 10, an Internet 105, or 
another content data source. Processing resources disposed in a computer can 
layout the content (step 140), e.g., according to the contends HTML description. 
During the layout 140, the processing resources generate rendering instructions 
140 A and derive a semantic representation 1408 of the multimedia content. 

25 Note that for some embodiments, layout 140 is not required. For these 

embodiments, the semantic representation 1408 can be generated fi-om play 
instructions, as shown in Figure 3 (at step 3 1 5). 

Content browsers can use the semantic representation 1408 to determine 
location, size, shape and targets of hyperlinks; and content play instructions. 

30 The semantic representation 1408 can also be used to describe other interactive 
presentation elements, e.g., HTML forms. The semantic features corresponding 
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to the depicted graphical representation generated for play when using 
traditional content browsers coupled with traditional caches 1 10 persist only as 
long as the content is being viewed. Because the semantic features must be 
present whenever the multimedia content is played, and because traditional 
5 caches 1 10 store the multimedia content in a non-rendered original form, 
traditional browsers must re-render the graphical representations each time a 
user requests the content, as shown in Figure 1. 

A render engine then renders the muUimedia content (at step 150) 
according to the rendering instructions 140 A to form rendered content 160 
10 (otherwise referred to herein as the rendered representation of the multimedia 
content). Finally, a multimedia play engine uses both the rendered content 160 
and the semantic representation MOB to play the rendered content (at step 170). 
For multimedia content including images, the playing 170 includes displaying 
the rendered image on a user screen according to the semantic representation 
1 5 MOB. The rendered content 1 60 is also referred to herein as a rendered 
representation of the multimedia content. 

In prior art implementations, playing 170 occurs after layout 140 (or 
other process in which the semantic representation MOB is generated), and 
rendering 150 have been completed. Rendering 150 and generation of the 
20 semantic representation MOB require a relatively long time between when the 
play request 120 is received and when multimedia content is played 170 
compared to the time required to play content using various method 
embodiments of the invention. Also, greater data processing is required for 
rendering 150 and generation of the semantic representation MOB for the 
25 multimedia content than the processing required for play 1 70 using various 
method embodiments of the invention. 

The details of retrieving (step 260), rendering 150, and playing 170 
multimedia content for some embodiments of the mvention are illustrated in 
Figs 2 through 4. Fig. 2 provides an overview of the retrieving 260, and the 
30 playing 1 70 processes for multimedia content, e.g., HTML content, using a 

rendered cache 201 . The methods can be unplemented in at least one computer 
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having one or more programs for retrieving and playing multimedia content. 
The benefits of using the rendered cache 201 for subsequent access to the same 
multimedia content are also described below. 

The rendered cache 201 includes not only rendered content 160 (which 
5 can include image data) but also some means of reconstructing the semantic 
representation HOB of the multimedia data. The reconstruction of the semantic 
representation 1403 can be done using proprietary image formats or separate 
files that describe the semantic features. This semantic representation MOB can 
include locations, sizes, and destinations of hyperlinks, descriptions of 
10 animations or other dynamic content, and other "meta" information. Meta 
information can include tagging, refresh (client pull replacement), Meta lists, 
and platform for Internet content selection (PICS) association labels. 

Some embodiments of the invention (including VirtualModem™ 
interactive presentation systems provided by Interactive Channel Technologies, 
15 Inc. located in London, Ontario, Canada) use an XML language called VMML 
to store the semantic content. The VMML semantic content can include markup 
to represent the following semantic features of the rendered content 160: 

1. Location, size, shape, and target indices (such as URI) of hyperlinks, 

2. Size and relative location of HTML frames in the rendered image, 
20 3, Size, location, and timing of animated GIFs, 

4. Size, location, and type of HTML form elements, 

5. Timing of multimedia content elements, and 

6. Other play 170 instructions. 

Proper use of content from a rendered cache 201 eliminates the steps of 
25 generating a semantic representation 140B, layout 140 (when needed), and 

rendering 150. On the other hand traditional web browsers using traditional 

caching mechanisms must perform these steps before playing 170 the content. 

Eliminating these steps reduces the time and use of processing resources 

required for playing 1 70 the multimedia content. 
30 A rendered cache 201 can include of two types of data objects: 

multimedia content and semantic content. The content can be stored 320 in any 
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format (i.e., the caching mechanism is independent of file format). Typically, 
the layout 140 and/or rendering 150 processing resources format the semantic 
presentation 1408 for storage in the rendered cache 201 as semantic content. 
Ahematively, the layout 140 and/or rendering 150 processing resources can 
5 transfer the semantic presentation MOB to rendered cache 201 server processing 
resources which then format the semantic presentation into semantic content to 
be stored in property indexed files for retrieval 260. For some embodiments of 
the invention, content browsers (and/or other client applications using content 
fi-om the rendered cache 201) can include processing resources, such as a 

10 program, for detecting the format of the rendered content 1 60 and for viewing 
multimedia content. 

When a request for content is received (step 210) the content browser 
can determine (step 220) whether a rendered representation of the content 
already exists in the rendered cache 201. The browser can also determine (step 

15 230) whether the content in the rendered cache 201 is outdated. The content 
request received at step 210 can be provided to the browser using a file target 
index, such as a Universal Resource Indicator (URI), 

Once it has been determined whether an updated rendered representation 
of the requested data already exists in the rendered cache 201, a first and 

20 simpler processing path indicated in Figure 2 can be followed. Because the 
rendered cache 201 contains valid rendered content corresponding to the 
request, browser engine processing resources can simply read the semantic 
content and the rendered content 160, restore the semantic representation (step 
240), and then play 170 the rendered content corresponding to the requested 

25 content. 

The process proceeds along a second path if the server-based system (or 
other processing resources coupled with the rendered cache 201) has determined 
that the requested content is not in the rendered cache 201, or that the content 
stored in the rendered cache is outdated. Along the second path, the browser 
30 submits a request to retrieve the content ft"om an updated source (e.g., the 

Internet 1 05) and retrieves the content (step 260). After the updated content has 
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been stored 320 in the rendered cache 201 (as shown in Figure 3), the process 
continues along the first method path as long as the stored content does not 
become out of date. The first method path, as shown in Figure 2, includes 
reading the semantic content and the rendered content 160, and restoring the 
5 semantic representation (step 240), to play 1 70 the rendered content for each 
request - 

Figure 3 illustrates a more detailed depiction of the layout 140 and 
storing 320 processes. After retrieving 260 the requested multimedia content 
with layout instructions and/or play instructions, the computer determines 
1 0 whether layout 140 is required for the muhimedia content (step 3 1 0). The 
semantic representation 1408 of the semantic features is generated during the 
layout 140 process, or generated from play instructions (step 315) when no 
layout is required. 

After rendering 150, the rendered content 160 is stored 320 in the 
1 5 rendered cache 201 . Similarly, after construction of the semantic representation 
MOB, the semantic representation is formatted as semantic content and also 
stored 320 in the rendered cache 201. 

If the rendered cache 201 stored only the resulting rendered content 160, 
the description of the hyperlinks, display instructions and other semantic 
20 content would be lost. The semantic content can take the form of flat text files, 
XML or other structured files, or other proprietary formats. Some embodiments 
of the invention format the semantic content according to an XML language 
called VirtualModem™ Markup Language (VMML) to represent the semantic 
features of HTML pages and TVML presentations. The rendered content 160 
25 and semantic content can be stored in a traditional cache, a database, a file 

system or other storage media. The underlying file system can be used to store 
the content in a directory and file hierarchy that represents the rendered cache 
201. 

The rendered content 160 stored 320 in the rendered cache 201 can 
30 include images, audio, text, fiill motion video, animations, etc. The content is 
stored in the rendered cache 201 regardless of its format [i.e. the rendered cache 
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201 can store binary large objects (blobs) or format-independent objects]. The 
format in which the semantic content is stored is independent of the rendered 
cache 201 mechanism. The content browsers and other client applications that 
access the rendered content 160 stored in the rendered cache 201 include 
5 processing resources adapted to recognize the format and interpret the semantic 
content appropriately. 

According to some embodiments of the invention, content browsers and 
other client applications include processing resources to recognize and play 170 
the rendered content 160 after the corresponding format-independent objects are 

1 0 retrieved 260 from the rendered cache 20 1 . Some embodiments of the 

invention, including various VirtualModem™ presentation systems, can render 
HTML pages into a proprietary image format, called a fat macroblock (FMB). 
that is suitable for display on televisions. FMB's are described in greater detail 
by United States patent application serial number 09/287,235, entitled "System 

1 5 and Methods for Preparing Multimedia Data Using Digital Video Data 

Compression", fded April 6, 1999, having inventors Antoine Boucher, Paul E. 
McRae, and Tong Qiu, the entire contents of which are hereby incorporated 
herein by reference as if fully set forth herein. 

In the case where the content is not missing but is outdated, the entire 

20 content can be retrieved 260, or just the outdated portions can be retrieved. By 
retrieving 260 only outdated portions some savings can be gained in the 
rendering 150 step by eliminating the need for a full rendering. For example, 
perhaps only an animated image on an HTML page has changed in the 
requested content. The rendering system can detect this situation and render 

25 1 50 only the new animation rather than the entire page. 

Once the needed portion of the request content has been retrieved 260, 
the content is rendered 1 50 before it is played 1 70. The retrieved content is 
handed to a rendering system that typically performs the following actions: 

1. Laying out 140 of the content according to the appropriate rules (e.g., 
30 HTML rules) 

2. Rendering 1 50 the content according to the rendering instructions 
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140 A, thereby producing presentation data (e.g., for an MPEG image formatted 
as an FMB- or set of images for HTML frames) that represent the fully rendered 
representation of the content (e.g., the HTML page). The page may also have 
other graphical elements created for such things as animated GIFs. 
5 3. Generating 315 a semantic representation MOB of the semantic 

features. Generally, the layout engine or the render engine creates the semantic 
representation MOB from the layout 140 or play instructions. For an HTML 
page, the semantic representation MOB can include the location, size, shape, 
and target of all HTML anchors (links to other HTML pages), the timing, 

10 location, and size of animated GIF frames, the size and relative location of 
HTML frames, information on HTML forms that can be accessed from the 
page, and HTML meta values. 

4. Storing 320 the rendered content 160 [e.g., MPEG image(s)] in the 
rendered cache 201 using an appropriate index, e.g., a URI. The semantic 

15 content is also stored 320 in the rendered cache 201 using an appropriate index. 
In some embodiments, the semantic content can be stored 320 in an XML- 
based format so that it can be easily parsed and restored (e.g., in step 240) in the 
fiiture. After the rendering system is finished, the rendered content 160 can be 
provided to the user by simply reading and restoring 240 and playing 170 the 

20 content. 

The "format" of the semantic representation MOB is determined by the 
engine that generates the semantic representation (e.g. Netscape Communicator 
and Microsoft ® Internet Explorer use the DOM). This internal semantic 
representation MOB is then stored as a physical entity (semantic content) in the 

25 rendered cache 201 . The format of semantic content is adapted for the browser 
engine that reads the semantic content for play 170. The format of the semantic 
content is sufficiently detailed for the browser engine to create its own semantic 
representation MOB. The semantic representation MOB in the browser engine 
can be the same internal format that the layout/render engine uses or the 

30 semantic representation in browser can have a different format. 

As shown in Figure 4, when a request is received for content already in 
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the rendered cache 201 the rendering system process can be skipped entirely. 
The following simple steps are all that is involved to play 1 70 content already in 
the rendered cache 201 . 

1. Read the semantic content and the rendered content 160, and restore 
5 the semantic representation HOB from the semantic content stored in the 

rendered cache 201, e.g., the VMML description. 

2. Play 170 the rendered content 160 on the user's screen according to 
this semantic representation HOB. 

Some multimedia content, such as an HTML web page, does not fit 
10 entirely on a user's screen at once. For such partial page displays, the browser 
can use the semantic representation HOB to determine which portion of the 
page should be displayed, and for some embodiments which subset of the 
hypertext links are selectable on the page portion. An example of this scrolling 
is described below in the "HTML Page with Layout" example below. 

15 

Retrieving Content from the Rendered Cache 

When a content browser, or other client application, requests a target 
index, such as a URI, the rendered cache 201 mechanism first looks in the 
rendered cache for a rendered representation of the content. The caching 

20 mechanism provides a means to search and retrieve this content based on the 
content's indexing indication. Examples of cache retrieval mechanisms include 
database queries, simple index files, file system directory structures, or 
traditional browser caches. 

If the rendered content 160 can be found in the rendered cache 201, the 

25 content will be displayed very quickly. The semantic representation HOB of 
the rendered content 160 will be restored using the semantic content stored in 
the rendered cache 201 (i.e. the semantic features need not be computed again 
before the rendered content is played). For example, some embodiments restore 
the semantic representation HOB of a rendered HTML page by reading the 

30 VMML formatted semantic content. 

If the content browser cannot locate a rendered representation of the 
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multimedia content in the rendered cache 201, or the browser determines that 
the content is out of date, then the content can be retrieved 260 (either from a 
traditional cache 110, from the Internet 105, or from another content source) and 
rendered 150. The retrieval 260 and rendering 150 results in at least one new 
5 rendered cache 201 entry that can be used the next time the multimedia content 
is accessed. 

A system that uses a rendered cache 201 will, after determining that no 
rendered representation is in the cache, perform the same steps as described 
above. That is, the HTML source will be read and the page laid out 140. The 

10 resulting rendering instructions 140A are followed but rather than displaying the 
page (or, alternatively, in addition to displaying) the page the rendering will be 
stored as a graphical image in the rendered cache 201. The semantic content 
(describing the location, size, and target URI of the single hyperlink on the 
image) is also stored in the rendered cache 201, The next time and every 

15 subsequent time the browser receives a request to view this URI, the browser 
simply reads the semantic content and the rendered content 160, restores the 
semantic representation HOB, and displays the rendered content. Thus, the use 
of the rendered cache 201 saves the cost of processing for layout 140, 
generation 315 of the semantic representation 1408, and rendering 150B. For 

20 more complicated HTML pages this savings can be substantial. 

Examples 

Specific embodiments of the invention are further described by the 
25 following, non-limiting examples which will serve to illustrate in some detail 
various features of significance. The examples are intended merely to facilitate 
an understanding of ways in which the invention may be practiced and to further 
enable those of skill in the art to practice the invention. Accordingly, the 
examples should not be construed as limiting the scope of the invention. 

30 

An HTML Page with Layout using a Distributed Server-Based Content 
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System 

Some embodiments of the invention provide storage 320, retrieval 260 
and/or play 170 of HTML pages. One embodiment of the invention is 
represented by the Toronto Stock Exchange (TSE) HTML homepage illustrated 
5 by Figures 5 A and 5B. 

For this embodiment, the "content browser" can be broken up into a 
distributed server-based content preparation and viewing system. The viewing 
system can include a display device, e.g., a television, and a digital set-top box 
(such as a General Instruments DCT-2000). 

10 For some HTML page embodiments, the set-top box has neither the 

processing nor the storage resources needed to render 150 or cache content. The 
set-top box typically does have the capability to decode and play MPEG images 
and Dolby AC-3 audio, and some limited graphics capabilities in order to do 
text and simple graphical overlays. For these embodiments, all access to 

1 5 rendering 1 50 processing resources and content stored in the rendered cache 20 1 
is done at the server. These embodiments are described in greater detail in the 
"System" section below. 

In other embodiments, the set-top box, or other addressable processing 
equipment, can have processing resources and storage medium capable of 

20 rendering 1 50 and caching the content. In response to the server-based system 
receiving a request to view some content with the URI http://www.tse.com/ and 
determining that the content is either not in the rendered cache or is outdated, 
the server system browser requests retrieval of the TSE web page and any 
graphical elements the TSE web page references. 

25 Once the web page and graphics have been retrieved 260 (either from a 

traditional cache 1 10 or from the Internet 105) the browser requests that the 
content be laid out 140 and rendered 150. The rendering system creates an 
MPEG representation (in FMB format) of the rendered web page. Because 
MPEG is the only image format the GI DCT-2000 recognizes, we use MPEG in 

30 this example. The rendering system can also generate other FMB files 

representing animated GIF frames, if animated GIFs were referenced in the 
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HTML page. The rendering system also creates a semantic representation HOB 
of the page including the location, shape, size, and target of all hyperlinks; 
location, size, and timing of animated GIF frames; HTML form information; 
and HTML meta information. 

The FMB files are stored 320 in the rendered cache 201 using the URI 
of the HTML page ("www.tse.com") as an index. The semantic content is also 
stored 320 in the rendered cache 201 using the URI as an index. The semantic 
content is stored in an XML format called VMML. For distributed systems 
embodiments, e.g., the VirtualModem™ system, the internal semantic 
representation 1408 for the layout/render engine is different than the semantic 
representation HOB for the browser engine (although these semantic 
representations HOB are conceptually equal). The stored semantic content (in 
the form of VMML for VirtualModemTM) is detailed enough to allow for 
"information transfer" so that two different semantic representations HOB can 
be used. 

Once the rendered content 160 (FMBs) and semantic content (VMML) 
are stored 320 in the rendered cache 201 , the browser can then read and restore 
the semantic representation HOB based on the VMML file. Using this semantic 
content the web page can be displayed. 

The first screen capture (Fig. 5 A) of the TSE homepage shows the top 
portion of the page. The rectangular highlight box 510 in the top left comer 
indicates that the user can select the fu-st hyperlink for viewing. Users can press 
arrow keys on their remote control to move fi-om one link to another link on the 
page. The browser provides enough information for the set-top box to draw the 
highlight box 510 and to navigate the page fi-om link to link using the arrow 
keys. 

Eventually the user may scroll past the bottom of the screen. The set-top 
will then inform the server-based browser that a scroll is required and the 
browser will then determine firom the semantic content which new portion of the 
rendered MPEG should be visible and which new subset of the hyperlinks is 
now selectable. 
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The second screen capture (Fig. 5B) illustrates the TSE homepage after a 
scroll down. The user can continue to view the same page scrolling around and 
viewing the content in the fashion described above. However, once a link is 
selected the browser is informed of the corresponding new URI request and the 
5 retrieval 260 (or read and restore 240) process are initiated again after the 
browser receives a play request 120. 

An HTML Page with Layout using a Self-Contained Content System 
Some embodiments do not use the distributed server-based content 

1 0 preparation and viewing system described in the above example (HTML Page 
vnth Layout). Instead, these embodiments are self-contained content systems 
vwth layout 140, rendering 150, and play 170 processes all combined in a single 
computer program. Netscape Communicator and Microsoft ® Internet Explorer 
are examples of such embodiments. 

1 5 Such "traditional" browsers can also use the invention to reduce retrieval 

260 and playing 170 time, and decrease processor usage. Traditional web 
browsers have long used caching technologies to minimize the need to use 
slower content retrieval 260 methods such as network access. These browsers 
store the original retrieved content in a cache database. When a request to view 

20 content is received, the browser searches the traditional cache 1 1 0. If the 

content not in the cache then the browser retrieves the content from an alternate 
source (such as the Internet 105). Visiting web sites that reside in a rendered 
cache 201 results in almost instantaneous display of the web site content rather 
than the usual delay (due to the cost of layout, rendering and creation of 

25 semantic context) that is normally seen. 

Whether or not the content was found in the traditional cache 1 10, the 
content is then read and laid out 1 40 according to the rules of HTML. Laying 
out 140 produces rendering instructions 140 A and a semantic representation 
MOB of the content. The page is then rendered 150 to a graphical format 

30 (typically a bitmap) and played 170 according to the semantic representation 
MOB. These steps are performed each and every time the content is requested. 
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For the self-contained embodiments of the invention, when a request for 
content is received the browser will search in the rendered cache 201 to 
determine whether a rendered representation of the content is available. If the 
content is not in the rendered cache 201, or if the rendered content is found to be 
5 outdated, then the content must be requested from an alternate source (such as 
the Internet 105, or a traditional cache 1 10). Once the content is received it will 
go through the same layout 140, rendering 150, and generation 3 1 5 of the 
semantic representation MOB steps as these browsers do now. 

The difference is that once the rendering 150 and generation 3 15 of the 

10 semantic representation HOB is complete the rendered content 160 and the 
semantic representation HOB are stored in the rendered cache 201. 

Once the content is stored in the rendered cache 201, then each time the 
browser receives a request for this content, the browser simply reads and 
restores 240 the semantic representation HOB and plays 170 the rendered 

15 content 160 according to this semantic representation. The format of the 

rendered content and semantic representation are entirely up to the browser. It 
is recommended that the rendered content be stored in a "native format". That 
is, a format that the browser can immediately recognize and does not have to 
convert to a recognized format. It is also recommended that the format for the 

20 semantic representation HOB be rich enough to cover all the various semantic 
elements that HTML can describe. VMML is a good example of such a format. 
For self-contained systems the format of the internal semantic representation 
HOB is likely to be the same for both the layout 140/render 150 and browser 
portions of the program. 

25 Another related embodiment that could benefit from the invention is 

what is conmionly referred to as a "web proxy". A web proxy is a computer 
program that retrieves content on behalf of content browsers. Various 
embodiments of the invention enable the web proxy to only retrieve 260 content 
from the Internet 1 05 for the first request, while all future requests for the 

30 content from browsers using the proxy use the locally cached version. 

Note that in either the distributed or self-contained scenarios, the task of 
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converting to and from the stored semantic content format is up to the relevant 
engines (layout 140/render 1 50 engine for storing and browser engine for 
retrieval 260. 

An alternate scenario could involve the layout/render engine transferring 
5 the semantic representation (through some communications medium) to a 
"rendered cache server" that converts the representation into semantic content. 
This server would also receive request to retrieve content from the cache and 
would read the semantic content, convert it to an appropriate internal 
representation and then transfer this representation. In this case the task of 
10 converting to and from semantic content is entirely up to the "rendered cache 
server". In 

practice, this approach is less flexible than alternative approaches. 

In the case where the web proxy and the content browsers all have 
access to the same storage or have access to a fast internal communications 

15 network the web proxy could perform the layout 140, rendering 150, and 

generation 315 of semantic representation 1406 steps on behalf of the content 
browsers. In such a scenario, when a content browser receives a request for 
content, the content browser can either look directly in the rendered cache 201 
or query the web proxy for the rendered content 160. The browser can then 

20 simply read and restore 240 the semantic content and display the rendered 
content 160 accordingly. This use of the web proxy allow for the use of very 
small and efficient web browser implementations since all the resources for 
layout 140, rendering 150, and generation 315 of the semantic representation 
MOB are external of the browser. 

25 An intelligent web proxy can pre-render the content that it downloads in 

order to offset the rendering cost in browsers. This approach is especially 
beneficial in situations where client computing resources are limited. A key 
application of this approach is in the emerging market of set-top devices and 
other network computers. These devices typically have tightly constrained 

30 resources and do not presently provide true web browsing. The use of a 

rendered cache proxy would offload the process of layout 140, rendering 150, 
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and generation of the semantic representation MOB. 

In some embodiments, word processing programs can store 320 
rendered documents in a rendered caciie 201 for faster loading and previewing. 
Using the rendered cache 201 for storing 320 word processing documents also 
5 enables programs other than the word processor to preview the content without 
using proprietary plug-ins or libraries. 

Multimedia Content with Play Instructions 

Some embodiments of the invention provide storage 320, retrieval 260 and/or 
10 play 170 of multimedia content. The multimedia content can include images, 
audio, text, graphics, and full motion video, all of which can be timed to play at 
different moments. This multimedia content can have a means of referencing 
other multimedia content in a manner similar to HTML hyperlinks. Some 
embodiments of the invention, including the VirtualModem™ system from 
15 Interactive Channel, use an XML language called TVML to represent the play 
instructions of a multimedia presentation, TVML can include markup to 
represent the following play instructions of the multimedia content: 

1. Timing of multimedia content playing; 

2. Order of multimedia content playing; 

20 3. Size and location of multimedia content; and 

4. Location, size, shape, and target URI (or other index) of hyperlinks. 
One embodiment of the invention is represented by the News Menu 
TVML presentation illustrated by Figs, 6A through 6D. Fig. 6A illustrates a 
timeline representing how the News Menu TVML presentation should be 
25 played. Figs. 6B through 6D show the images that make up the News Menu 
TVML presentation. As in the previously described embodiment (HTML with 
layout) the "content browser" can be broken up into a distributed server-based 
content preparation and viewing system. 

The server-based system can receive a request to view some content 
30 with the URI httD://www. virtualmodem.com/news.tvml and then determined 
that the content is either not in the rendered cache 201 or is outdated. The 
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browser can respond to this circumstance by submitting a request to retrieve the 
TVML presentation and any muUimedia elements referenced by the 
presentation. Once the presentation and its multimedia elements have been 
retrieved 260 (either from a traditional cache 1 10 or from the Internet 105), the 
5 browser requests that the content be rendered 150. In this case, layout 140 is 
unnecessary and the rendering 150 can be limited to converting the mukimedia 
content into a format that the set-top recognizes. In the case of the GI DCT- 
2000 images and full motion video are converted to MPEG formatted data and 
audio is converted to Dolby AC-3 formatted data. 

1 0 The rendering system can also generate 315a semantic representation 

140B of the page from the TVML play instructions. The semantic 
representation 1408 can include context such as the relative play times and 
order of the multimedia content; the location, shape, size, and target of all 
hyperlinks; and TVML meta inforaiation. The rendered content 160 can be 

15 stored 320 in the rendered cache 201 using the URI of the presentation 
("www.virtualmodem.com/news.tvml") as an index. The appropriately 
formatted semantic content based on the semantic representation HOB is also 
stored in the rendered cache 201 using the URI as an index. For some 
embodiments of the invention, the semantic content is stored in a VMML 

20 format. 

Once the rendered content 160 and semantic content (VMML) are stored 
in the rendered cache 201, the browser can read and restore 240 the semantic 
representation 1403 from the VMML file in which the semantic content is 
disposed. Using this semantic representation 1403 the presentation can be 
25 displayed. 

Figure 6A shows the start time and duration that each image of the 
presentation should be played, and illustrates the start time and duration of the 
accompanying audio. The presentation plays 170 from to to ty 

Figures 6B through 6D show each of the images used in the for the 
30 News Menu TVML presentation. The first image 650 of the presentation, 

shown in Fig. 63, includes a single circle with a small diameter around the top 
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of the transmitter to indicate that a signal is being sent from a transmitter. As 
shown by the first time line 610, the first image 650 is shown from to to t,. 

The second image 660 of the presentation, shown in FIG. 6C, includes 
three circles around the top of the transmitter to indicate that the signal will be 
5 received by the user sooner than when the first image 650 was displayed. As 
shown by the second time line 620, the second image 660 is shown from t, to tj. 

The third image 670 of the presentation, shown in FIG. 6D, includes a 
first hyperlink that is enclosed by a rectangular highlight box 5 10 to indicate 

1 0 that the first hyperlink "World News Update" is presently available for 

selection. As shown by the second time line 630, the third image 660 is shown 
from tj to ty A user can press arrow keys disposed on the user's remote control 
devices to move from link to link in the third image 670. The browser provides 
enough information for the set-top box to draw this rectangle and to navigate 

15 using the arrow keys from link to link. If the multimedia content is larger than 
the physical screen then it becomes possible to scroll in the same maimer as 
described in the "HTML with layout" example. As shown by the fourth time 
640, the accompanying audio plays 170 for the entire duration of the News 
Menu TVML presentation. 

20 In the News Menu TVML presentation example only the third image 

670 of the presentation contains hyperlinks. However, in other embodiments of 
the invention, any of the earlier images can also contain hyperlinks. The 
browser can update the client (set-top box or other addressable processing 
equipment) whenever the semantic representation MOB (e.g. hyperlink 

25 information or image display duration) changes. 

The presentation can play 170 until all multimedia objects have been 
played. The user can continue to view the last image of the presentation in the 
same manner as for HTML pages. The user can also manipulate the remote 
control VCR fimctions to rewind, fast-forward, or pause the presentation. 

30 However, once a hyperlink is selected the browser will be informed of the new 
URI request and the content retrieval process will start again with a request for 
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content. 

Systems for Storing, Retrieving and Playing Multimedia Content 

Some embodiments of the invention include systems for storing 320, 
5 retrieving 260 and playing 1 70 multimedia content using a rendered cache 20 1 . 
Listed below are the key elements of a system that can implement various 
embodiments of the invention. Previous descriptions and examples mentioned 
in the "Distributed Server-Based Content System" section have illustrated the 
use of the invention in a distributed server-based system. In such a system the 
10 various complimentary components, such as those listed below, are typically 
found in separately running processors that can reside in a single computer or in 
multiple connected computers. Some embodiments, such as the 
VirtualModem^ system can include the following components: 

Web crawler processing resources adapted to access multimedia content 
1 5 from source data storage. The multimedia data can include HTML and TVML 
content. The source data storage can include at least one of the Internet 105 and 
a web proxy cache. 

Rendering processing resources adapted to generate semantic 
representation MOB of, and render 150 multimedia data, and can format the 
20 semantic representation as semantic content. In some embodiments, a rendering 
program can also be adapted to layout 140 the multimedia data. 

Multimedia playing processing resources, such as an audio/video 
terminal server (AVTS), adapted to play multimedia content. Such play can 
include displaying images and playing audio and full motion video. Some 
25 embodiments of an AVTS are described in greater detail in United States patent 
application serial number 09/255,052, entitled "System and Method for 
Interactive Distribution of Selectable Presentations," filed February 22, 1999, 
and having inventors: Antoine Boucher, James Lee Fischer, and Allan E. 
Lodberg, the entire contents of which are hereby incorporated herein by 
30 reference as if fully set forth herein. 

Browser processing resources adapted to interpret the semantic content 
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and control when and how the multimedia content should be played. The 
browser processing resources can act as the "control center" for the entire 
process. The browser processing resources can communicate with the web 
crawler, rendering, and the multimedia playing processing resources and 
5 coordinate the interactions of each of these. 

A second group of embodiments discussed above in the "Self-Contained 
Content System" section contain all the required rendered cache 201 
complimentary components in a single program. Netscape Communicator and 
Microsoft ® Internet Explorer are both examples of such single programs. Both 

1 0 of these browsers include programs adapted to retrieve 260 content from the 
Internet 105 or a proxy. Netscape Communicator and Microsoft ® Internet 
Explorer also have layout 140, rendering 150, and semantic representation 1408 
generating capabilities. They both have the ability to display the rendered 
content 160 to a computer monitor display and they both can interpret the 

1 5 semantic representation 1408, 

FIG. 7 illustrates the components and features configured in a system for 
accessing multimedia content using a rendered cache 700 representing one 
embodiment of the invention. The system for accessing multimedia content 
using a rendered cache 700 includes the components and features described 

20 below, including: access to source content 71 0, at least one layout engine 720, 
at least one render engine 730, a rendered cache 201, at least one multimedia 
play engine 750, at least one browser engine 760, and a display 770. These 
components can be combined together to form one or more computer programs 
that implement the storing 320, retrieving 260 and playing 1 70 methods 

25 described above. 

Source content is content that is not yet rendered. The source content 
can include HTML, XML, images, audio, text, and fiill motion video. Access to 
source content 710 can be through an Intranet, the Internet 105, a web proxy, or 
on local storage. Connections adapted to provide such access can be through 

30 any carrier capable of providing sufficient bandwidth for practical retrieval 260 
the content, such as: digital subscriber line (DSL), cable modem, T-1, T-2, T-3, 
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OC-1 through OC-256, fiber distributed data interface (FDDI), El through E5, 
Ethernet, fast Ethernet, and Gigabit Ethernet. Access to source content 710 can 
also include processing resources adapted to use standard Internet protocols 
such as TCP/IP and HTTP, and to read files from a file system. The component 
5 providing access to source content 710 includes processing resources for 

retrieving the source content, such as the content fetch 715 resources shown in 
FIG. 7. 

The system for accessing multimedia content using a rendered cache 700 
can include layout processing resources, such as a layout engine 720, adapted to 

10 derive rendering instructions 140A from a content definition (e.g., HTML). The 
layout engme 720 can also derive a semantic representation MOB of the features 
of the content fi-om the layout 140, or fi-om the play 170 instructions. Netscape 
Communicator and Microsoft ® Internet Explorer both contain processing 
resources to perform HTML layout 140 as part of their overall fimctionality. 

1 5 Stand-alone layout engines 720 include Spyglass Device Mosaic, NGLayout 
fi-om Mozilla, and Chimera. In some embodiments, processing resources other 
than the layout processing resources can be adapted to generate 315 the 
semantic representation 1408 fi-om play 170 instructions. 

The system for accessing multimedia content using a rendered cache 700 

20 can include rendering processing resources, such as a render engine 730, 

adapted to create a graphical representation of content that has been laid out 140 
by the layout engine 720. The rendering engine 730 can also have the capability 
of converting content that does not require layout 140 into a form that is ready 
for rapid play 170. 

25 Many layout engines 720 also include a render engine 730. Systems that 

have layout engines 720 that do not include a rendering 150 capability have a 
separate render engine 730 and typically specify the interface that a rendering 
engine must have (e.g., this is hov^r Spyglass Device Mosaic works). Both 
Netscape Commimicator and Microsoft ® Internet Explorer include rendering 

30 engines as part of their overall fimctionality. Both of these browsers render 1 50 
the content into a bitmap that can be displayed on a computer monitor display 
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770. Some embodiments, such as the VirtualModem™ system, use their own 
custom render engine 730 that renders the content to MPEG files stored in FMB 
format. 

The rendered cache 201 provides access to an indexed storage 
5 mechanism. The rendered cache 201 stores both the rendered content 160 and 
the semantic content so that these data objects can be easily retrieved 260 at a 
later time. The rendered cache 201 includes as indexing mechanism that can 
take a variety of forms including database queries, index files, file system 
directories. 

10 The format of the rendered content 160 is independent of the storage 

mechanism. A format that requires little or no conversion at play 1 70 time (i.e. 
a "native format") provides greater time and processing savings using the 
rendered cache 201. 

The format of the semantic content is also independent of the storage 

1 5 mechanism. The semantic content format used in the system for accessing 
multimedia content using a rendered cache 700 that fully captures all the 
semantic features of the rendered content 160 provides enhanced play results. 
The semantic content format can avoid unneeded complexity to ensure that the 
processing and time required to restore the semantic representation 1408 are 

20 less than that required to layout 140 and re-render the content. 

The system for accessing multimedia content using a rendered cache 700 
includes multimedia play processing resources, such as a multimedia play 
engine 750, adapted to play the rendered content 160 on a display 770 device. 
The multimedia play engine 750 can read the rendered content 160 directly from 

25 the rendered cache 201 indexed storage mechanism, read the rendered content 
from memory, or otherwise receive the rendered content from an external 
source. Netscape Conununicator and Microsoft ® Internet Explorer both 
contain, as part of their overall functionality, processing resources to display 
multimedia content to a computer display 770. Some embodiments, including 

30 the VirtualModem™ system, include a separate program that is part of the 

overall distributed system, called the AVTS, that is adapted to play multimedia 
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content to APEs. 

The system for accessing multimedia content using a rendered cache 700 
also includes a browser engine 760 adapted to interpret the semantic 
representation 140B of the rendered content 160 being played 170. The browser 
5 engine 760 can read the semantic content directly from the rendered cache 201 
indexed storage mechanism, or interpret the rendered content 160 from memory, 
or otherwise receive the semantic content from an external source. 

The browser engine 760 can be adapted to interpret the semantic features 
from the semantic content. In some embodiments, the browser engine 760 is 

10 adapted to control navigation of hyperlinks (i.e. determining from user input 
which content should be displayed next). The browser engine 760 also can 
determine which portions of the rendered content 160 should be played 170, and 
which corresponding portions of the semantic representation MOB are active 
(e.g., when scrolling an image). 

15 The browser engine 760 can be included in commercially available 

software such as Netscape Communicator, Microsoft ® Internet Explorer, or 
any other browser engine that is adapted to perform the functions described 
above. Netscape Communicator and Microsoft ® Internet Explorer both 
contain, as a part of their fimctionality, processing resources adapted to interpret 

20 a semantic representation HOB [or Document Object Model (DOM) as both call 
it]. Both of these browsers use the DOM to determine which links are currently 
visible (and which others are scrolled off the screen), animated GIF timing and 
location, information about HTML forms and other HTML features. Some 
embodiments, such as the VirtualModem™ system, include a browser program 

25 that coordinates the retrieving 260 of content, layout 1 40 and rendering 1 50 of 
content, and playing 170 of rendered content. These browser embodiments can 
also contain processing resources for reading semantic content from the 
rendered cache 201 and restoring the semantic representation 140B. 

The above engines (layout, render, play, and browser) are all at least 

30 loosely coupled. That is, they need not be part of the same program but there 
needs to be some form of communication between them all. This 
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communication can take a variety of forms including inter-process 
commimication (such as shared memory, pipes, or messaging protocols), or 
shared files. Some embodiments, such as the VirtualModem™ system, use a 
communications protocol built on a user data protocol (UDP) to communicate 
5 between the various engines. Netscape Communicator and Microsoft ® Internet 
Explorer include all the engine components in the same program. 

There is no requirement that any of the above system components be 
directly tied together (i.e. included in the same program). However, there are 
advantages to tightly coupling certain components. For example, it is more 

10 efficient to couple the layout engine 720 and the render engine 730 in the same 
program. In such a scenario the rendering instructions 140 A resulting from 
layout 140 process can be used directly by the rendering engine 730 component. 
If the layout engine 720 and the render engine are separate programs, then 
some intermediate form of rendering instructions (e.g. either a file or data 

15 passed over a network) would have to be used. 

Only the component responsible for accessing source content 710 needs 
to include processing resources to access the communications carrier and the 
underlying communications protocol. It is not required that the other engine 
components have these processing resources. 

20 The layout engine 720 and the render engine 730 has access to the 

rendered cache 201 storage mechanism since they read the rendered content 160 
and the semantic content. 

The multimedia play engine 750 has access to at least the rendered 
content 160 portion of the rendered cache 201 storage mechanism. The browser 

25 engine 760 has access to at least the semantic content portion of the rendered 
cache 201 storage mechanism. Both the multimedia play engine 750 and the 
browser engine 760 can have fiill access to the entire rendered cache 201 storage 
mechanism but at minimum they have access to their respective content. 

Splitting access to the rendered content 160 and the semantic content 

30 allows for efficient distribution of the multimedia play engine 750 and browser 
engine 760. 
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Formatting the Semantic Content 

Some embodiments of the invention use an extensible markup language 
(XML) language to format and store 320 semantic content in the rendered cache 
5 201 . Embodiments including the VirtualModem™ system use a markup 
language called VMML to format and store 320 semantic content in the 
rendered cache 201 . 

VMML contains elements to describe the semantic features of both 
HTML and TVML. TVML is another XML language originally based on 
1 0 synchronized multimedia integration language (SMIL) from the World Wide 
Web Consortium at httD://www. w3.org/ . The descriptive elements include: 

1. Multimedia elements - The <img>, <audio>, <video>, and <text> 
elements are used to describe fully rendered multimedia objects. The <screen> 
element is used to describe fully rendered HTML. Each of these elements can 

1 5 include an optional start time using the "begin" attribute. 

2. Aggregation elements - The <par> and <seq> elements are used to 
describe how the multimedia elements are played. Elements inside a <par> are 
played in parallel. The start times of multimedia elements in a <par> are 
relative to the beginning of the <par>. Elements inside a <seq> are played 

20 sequentially. The start times of multimedia elements in a <seq> are relative to 
the end of the previous element. Both the <par> and <seq> elements can define 
optional start times using the "begin" attribute. 

For example, the following <par> element contains an <audio> and two 
<img> elements which are played in parallel (i.e. at the same time). The display 
25 of the second image is delayed by 5 seconds. 
<par> 

<audio src="voice-over.ac3"/> 

<img src="first-screen.finb"^ 

<img src="second-screen.ftnb" begin="5.0s"/> 

30 <ypar> 
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3. HTML elements - The <screen> element is used as a container for all 
the semantic information concerning a rendered HTML page. Elements allowed 
in a <screen> element include: 

<frame> - contains attributes for defming the FMB (the rendered frame), size, 
5 and location relative to other frames of the HTML page; 

<anchor> - each <fiBme> element can contain a list of <anchor> elements 
which describe the location, size, shape, and target of HTML hyperlinks; 
<form> - each frame can contain form elements which fully describe HTML 
forms; 

10 <animation> - <frame> elements can contain animation elements that describe 
the timing, size and location of animated GIFs. 

4. Non-display elements - The <title> and <meta> elements describe 
non-audiovisual features of the content. Examples of <meta> information 
include HTML refreshes, and expire metas. 

15 5. Anchors - Information about non-HTML hyperlinks is also described 

in VMML <anchor> elements. 

6. Applets - The <applet> element instructs the browser to run other 
applications. 

VMML contains other minor elements and a wide variety of attributes 
20 but the above list describes the major features. VMML is capable of describing 
all the various features of TVML and HTML in sufficient detail that the 
semantic representation HOB can be reconstructed after reading the semantic 
content from the rendered cache 201 . The reconstruction of the semantic 
representation MOB includes simple tokenization (i.e. text parsing) using freely 
25 available tools such as sgml-lex (available from http://wv^^.w3.org/^ . The 
parsing process is much faster and uses far fewer processor resources than the 
processes of layout 1 40 and rendering 1 50. 

The term coupled, as used herein, is defined as connected, although not 
necessarily directly, and not necessarily mechanically. The term substantially, 
30 as used herein, is defined as approximately (e.g., preferably within 1 0% of, 
more preferably within 1% of, most preferably within 0.1% of). 
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Advantages of the Invention 

A rendered cache 201 coupled with multimedia content render, play, and 
browser processing resources, representing an embodiment of the invention, can 
be cost effective and advantageous for at least the following reasons. The 
5 rendered cache 201 enables the play 170 of multimedia content in less time and 
using less data processing because the steps of layout and rendering are 
eliminated. 

All the disclosed embodiments of the invention described herein can be 
realized and practiced without undue experimentation. Although the best mode 
10 of carrying out the invention contemplated by the inventors is disclosed above, 
practice of the invention is not limited thereto. Accordingly, it will be 
appreciated by those skilled in the art that the invention may be practiced 
otherv^se than as specifically described herein. 

For example, although the rendered cache 201 described herein can be a 
15 physically separate module, it will be manifest that the rendered cache 201 can 
be integrated into the apparatus with which it is associated. Furthermore, all the 
disclosed elements and features of each disclosed embodiment can be combined 
with, or substituted for, the disclosed elements and features of every other 
disclosed embodiment except where such elements or features are mutually 
20 exclusive. 

It will be manifest that various additions, modifications and 
rearrangements of the features of the invention may be made without deviating 
from the spirit and scope of the underlying mventive concept. It is intended that 
the scope of the invention as defined by the appended claims and their 
25 equivalents cover all such additions, modifications, and rearrangements. The 
appended claims are not to be interpreted as including means-plus-function 
limitations, unless such a limitation is explicidy recited in a given claim using 
the phrase "means-for." Expedient embodiments of the invention are 
differentiated by the appended sub-claims. 
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CLAIMS 

What is claimed is: 

1 . A method, implemented in at least one computer, for storing 
5 multimedia data, comprising: 

detecting multimedia content, the multimedia content includes play 
instructions and at least one multimedia element, the at least one multimedia 
element includes at least one of graphical images, audio, text, and full motion 
video; 

10 generating a semantic representation of a rendered representation of the 

multimedia content from the play instructions, the play instructions include at 
least one of timing of the multimedia content and ordering of the multimedia 
content, the semantic representation describes at least one of: characteristics of 
the rendered representation, and relationships between different multimedia 

15 elements disposed in the rendered representation; 

storing the rendered representation of the multimedia content in a 
storage medium; and 

storing data corresponding to the semantic representation in the storage 
medium. 

20 2. The method for storing multimedia data of claim 1 , wherein the 

semantic representation includes at least one of: 

a location of a hyperlink; 

a size of the hyperlink; 

a shape of the hyperiink; 
25 a target index of the hyperiink; 

a size of a portion of the multimedia content disposed in a rendered 

image; 

a relative location of the portion of the multimedia content disposed in a 
rendered image; 
30 sizes of animated formatted graphics; 
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locations of the animated formatted graphics; 
timing of the animated formatted graphics; 
browser commands for addressable processing equipment; and 
data corresponding to the play instructions. 

3. The method for storing multimedia data of claim 1 , wherein: 
one of the at least one computer includes rendering processing resources 

adapted to generate the semantic representation; and 

storing the data corresponding to the semantic representation includes: 
formatting the semantic representation into semantic content to 
enable rapid restoring of the semantic representation by a content browser; and 
storing the semantic content. 

4. The meUiod for storing multimedia data of claim 3, wherem: 
storing the semantic content includes the rendering processing resources 

transferring a semantic content file to the storage medium; 

storing the rendered representation includes the rendering processing 
resources transferring at least one rendered representation file to the storage 
medium; and 

the method includes indexing the semantic content file and the at least 
one rendered representation file based on at least one universal resource 
identifier, to form at least one indexed file. 

5. The method for storing multimedia data of claim 4, including 
25 arranging the indexed files to enable a client application to retrieve the rendered 

representation and the semantic content, the client application includes the 
content browser. 

6. The method for storing multimedia data of claim 1 , including 
30 converting tiie multimedia content into the rendered representation of the 

multimedia content. 



15 
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7. A method, implemented in at least one computer, for storing 
multimedia data, comprising: 

detecting multimedia content including layout instructions, the 
multimedia content includes at least one multimedia element, the at least one 
5 multimedia element includes at least one of graphical images, audio, text, and 
full motion video; 

laying out the multimedia content according to the layout instructions to 
form rendering instructions and a semantic representation of a rendered 
representation of the multimedia content, the semantic representation describes 
10 at least one of: characteristics of the rendered representation of the multimedia 
content, and relationships between different multimedia elements disposed in 
the rendered representation of the multimedia content; 

rendering the multimedia content according to the rendering instructions 
to produce the rendered representation; 
1 5 storing the rendered representation in a storage medium; and 

storing data corresponding to the semantic representation in the storage 
medium. 

8. The method for storing multimedia data of claim 7, including: 
20 detecting play instructions, and 

wherein laying out includes incorporating the play instructions into the 
semantic representation, and the play instructions include at least one of timing 
of the multimedia content and ordering of the multimedia content. 

25 9, A method, implemented in at least one computer, for retrieving 

multimedia data, comprising: 

processing resources of a first computer of the at least one computer 
detecting a request for requested multimedia content; 

processing resources coupled with the first computer determining 
30 whether data corresponding to the requested multimedia content is disposed in a 
storage medium, the storage medium is coupled with the first computer, the 
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Storage medium includes rendered representations of multimedia content and 
semantic content, the semantic content includes data corresponding to semantic 
representations derived from one of: play instructions for the rendered content, 
and layout of the multimedia content, the semantic representations describe at 
5 least one of: characteristics of the rendered representations, and relationships 
between different multimedia elements disposed in the rendered representations; 
and 

responding to a determination that data corresponding to the requested 
multimedia content are disposed in the storage medium by: 
10 retrieving a rendered representation of the requested multimedia 

content; and 

retrieving semantic content corresponding to the requested 
multimedia content. 



15 10. The method for retrieving multimedia data of claim 9, including 

restoring the semantic representation for the requested multimedia content using 
the semantic content corresponding to the requested multimedia content. 

1 1 . The method for retrieving multimedia data of claim 9, wherein 
20 determining whether data corresponding to the requested multimedia content are 
disposed in the storage medium includes searching the storage medium using a 
retrieval mechanism adapted to search and retrieve content based on an index 
corresponding to the requested multimedia content. 

25 1 2. The method for retrieving multimedia data of claim 1 1 , wherein 

a semantic representation corresponding to the requested multimedia content 
includes play instructions for the requested multimedia content, and the method 
includes playing the requested multimedia content according to the play 
instructions. 

30 

1 3 . The method for retrieving multimedia data of claim 1 1 , 
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including: responding to a determination that data corresponding to the 
requested multimedia content are not disposed in the storage medium by: 

storing the data corresponding to the requested muhimedia content in the 
storage medium; 

5 retrieving the rendered representation of the requested multimedia 

content; and 

retrieving the semantic content corresponding to requested multimedia 
content. 

10 14. The method for retrieving multimedia data of claim 13 including 

restoring a semantic representation for the rendered representation of the 
requested multimedia content using the semantic content corresponding to 
requested multimedia content. 

15 15. The method for retrieving multimedia data of claim 9 including 

responding to a determination that data corresponding to the requested 
multimedia content is not disposed in the storage medium by: 

storing data corresponding to the requested multimedia content in the 
storage medium; 

20 retrieving the rendered representation of the requested multimedia 

content; and 

retrieving the semantic content corresponding to requested multimedia 
content, 

25 1 6. The method for retrieving muhimedia data of claim 1 0, wherein: 

data disposed in the storage medium includes at least one file; 
the at least on file mcludes the rendered representation of the requested 

multimedia content, and the semantic content corresponding to the requested 

multimedia content; 

30 restoring the semantic representation is performed by a client application 

using the semantic content corresponding to requested multimedia content; and 
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the method includes: 

a client computer requesting the requested multimedia content; 

and 

the client application recognizing and playing the requested 
5 multimedia content from at least a portion of the at least one file. 



17. The method for retrieving multimedia data of claim 9, wherein 
responsive to determining that data corresponding to the requested multimedia 
content are disposed in the storage medium, determining whether the data 
10 corresponding to requested multimedia content disposed in the storage medium 
require updating. 



18. The method for retrieving multimedia data of claim 9, including, 
prior to retrieving the rendered representation of the requested multimedia 

15 content: 

processing resources coupled with the first computer determining 
whether the data corresponding to the requested multimedia content disposed in 
the storage medium require updating: 

responsive to a determination that the data corresponding to the 
20 requested multimedia content disposed in the storage medium require updating: 

storing an updated version of the data corresponding to 
the requested multimedia content in the storage medium; 

retrieving at least a portion of an updated version of the 
rendered representation of the requested multimedia content; 
25 retrieving at least a portion of an updated version of the 

semantic content corresponding to the requested multimedia content; and 

restoring a semantic representation for the requested 
multimedia content using the at least a portion of the updated version of the 
semantic content. 

30 

19. The method for retrieving multimedia data of claim 1 8, wherein 



41 



wo 00/77664 



PCT/USOO/15938 



determining whether the data corresponding to the requested muhimedia content 
disposed in the storage medium require updating includes at least one of: 

comparing an expiry date tag for the data corresponding to the requested 
multimedia content disposed in the storage medium with a date corresponding 
5 to the updated version of the data corresponding to the requested multimedia 
content; and 

comparing the data corresponding to the requested multimedia content 
disposed in the storage medium with a version of data corresponding to the 
requested multimedia content disposed in a different storage medium, 

10 

20. The method for retrieving multimedia data of claim 9, including, 
prior to retrieving the rendered representation of the requested multimedia 
content: 

processing resources coupled with the first computer determining 
1 5 whether the data corresponding to the requested multimedia content disposed in 
the storage medium require updating; and 

responsive to a determination that the data corresponding to the 
requested muhimedia content disposed in the storage medium require updating: 
storing an updated version of the data corresponding to the 
20 requested multimedia content in the storage medium including an updated 

version of the rendered representation of the requested multimedia content, and 
an updated version of the semantic content corresponding to the updated version 
of the rendered representation; 

retrieving the updated version of the rendered representation of 
25 the requested multimedia content; 

retrieving the updated version of the semantic content 
corresponding to the updated version of the rendered representation; and 

restoring the semantic representation for the requested 
multimedia content corresponding to the updated version of the rendered 
30 representation using the updated version of the semantic content. 
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2 1 . The method for retrieving multimedia data of claim 9, wherein: 
the request includes an index corresponding to the requested multimedia 

content; and 

determining whether data corresponding to the requested multimedia 
5 content is disposed in a storage medium includes searching the storage medium 
using a retrieval mechanism adapted to search and retrieve content based on the 
index corresponding to the requested multimedia content. 

22. The method for retrieving multimedia data of claim 2 1 , wherein 
10 the retrieval mechanism includes using at least one of: 

a database query, 
index files, and 

a file system directory structure. 



15 23. A rendered cache comprising: 

a storage medium; and 

an indexing mechanism adapted to store and retrieve: 

a rendered representation of the multimedia content formatted for 
rapid play, the multimedia content includes at least one multimedia element, the 
20 at least one multimedia element includes at least one of graphical images, audio, 
text, and full motion video; and 

semantic content of the multimedia content, the semantic content 
includes data describing at least one of: characteristics of the rendered 
representation, and relationships between different multimedia elements 
25 disposed in the rendered representation. 



24. The rendered cache of claim 23, wherein the semantic content 
includes data corresponding to at least one of: 
a location of a hyperlink; 
30 a size of the hyperlink; 

a shape of the hyperlink; 
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a target index of the hyperlink; 

a size of a portion of the multimedia content disposed in a rendered 

image; 

a relative location of the portion of the multimedia content disposed in a 
5 rendered image 

sizes of animated formatted graphics; 
locations of animated formatted graphics; 
timing of animated formatted graphics; 
browser conunands for addressable processing equipment; and 
10 data corresponding to play instructions. 

25. The rendered cache of claim 24, wherein the play instructions 
include at least one of timing of the multimedia content and ordering of the 
multimedia content. 



15 



20 



26. The rendered cache of claim 23, wherein the indexing 
mechanism includes processing resources for converting the rendered content 
address into a unique index, the unique index is adapted to store and retrieve the 
rendered content and semantic content. 



27. A client comprising: 

processing resources adapted to detect a rendered representation of 
multimedia content and semantic content of the rendered representations, the 
multimedia content includes at least one multimedia element, the at least one 

25 multimedia element includes at least one of graphical images, audio, text, and 
full motion video, the semantic content includes data describing at least one of: 
characteristics of the rendered representation, and relationships between 
different multimedia elements disposed in the rendered representation; and 
processing resources adapted to respond to detecting the rendered 

30 representation of the multimedia content and the semantic content by playing at 
least a portion of the rendered representation according to the semantic content. 
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28. The client of claim 27, wherein the semantic content includes 
data corresponding to hyperlinks. 

29. The client of claim 27, wherein: 
5 the client includes a set-top box; 

only a portion of the multimedia content, and only a portion of the 
semantic content are received by the processing resources adapted to detect a 
rendered representation of multimedia content and semantic content of the 
rendered representations; and 
10 the semantic content includes data corresponding to scroll commands. 

30. A system for using multimedia content comprising: 

web crawler processing resources adapted to access the multimedia 
content from source data storage, the multimedia content includes at least one 
15 muhimedia element, the at least one multhnedia element includes at least one of 
graphical images, audio, text, and full motion video; 
rendering processing resources adapted to: 

generate a semantic representation of a rendered representation 
of the multimedia content, the semantic representation describes at least one of: 
20 characteristics of the rendered representation of the multimedia content, and 
relationships between different multimedia elements disposed in the rendered 
representation of the multimedia content; 

format the semantic representation as semantic content; and 
render the multimedia content into the rendered representation, 
25 the rendered representation is formatted for rapid play; and 
a rendered cache including: 
a storage medium; and 

an indexing mechanism adapted to store and retrieve: 

the rendered representation of the multimedia content; 

30 and 

the semantic content of the multimedia content; 
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31. The system for using multimedia content of claim 30 including: 
a browser adapted to: 

interpret the semantic content; and 

control play of the rendered representation of the multimedia 

5 content, and 

multimedia playing processing resources adapted to play the rendered 
representation of the multimedia content. 

32. The system for using multimedia content of claim 30, wherein: 

1 0 source data storage includes at least one of the Internet and a web proxy 

cache; and 

the rendering processing resources are adapted to layout the multimedia 
content. 

^ ^ 33 . The system for using multimedia content of claim 3 1 , wherein 

the browser processing resources: 

communicate with the web crawler processing resources, the rendering 
processing resources, and the multimedia playing processing resources; and 
are adapted to control the interactions of at least one of: 
20 the web crawler processing resources; 

the rendering processing resources, and 

the multimedia playing processmg resources. 

34. A system for accessing multimedia content comprising: 
25 a rendered cache including: 

a storage medium; and 

an indexing mechanism adapted to store and retrieve: 

a rendered representation of the multimedia content 
formatted for rapid play, the multimedia content includes at least one 
30 multimedia element, the at least one multimedia element includes at least one of 
graphical images, audio, text, and full motion video; and 
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semantic content of the multimedia content, the semantic 
content includes data describing at least one of: characteristics of the rendered 
representation, and relationships between different multimedia elements 
disposed in the rendered representation; and 
5 rendering processing resources adapted to: 

convert the multimedia content into the rendered representation, 
the rendered representation is formatted for rapid play; and 

create a graphical representation of the multimedia content. 

10 35. The system for accessing multimedia content of claim 34 

including: multimedia play processing resources adapted to: 
read the rendered representation; and 
play the multimedia content on a display; and 
browser processing resources adapted to interpret the semantic content. 

15 

36. The system for accessing multimedia content of claim 34 
including: 

layout processing resources adapted to: 

derive rendering instructions from a content definition; 
20 lay out the multimedia content; and 

generate a semantic representation of the multimedia content 
from lay out of the multimedia content; and 

wherein the rendering processing resources use the rendering 
instructions to create the graphical representation. 
25 1 

37. The system for accessing multimedia content of claim 35, 
wherein the browser processing resources are adapted to control play of the 
multimedia content. 

30 38. A method for playing multimedia content, comprising: 

retrieving a rendered representation of the multimedia content from a 
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Storage medium; 

retrieving semantic content of the rendered representation from the 
storage medium, the semantic content including data describing at least one of: 
characteristics of the rendered representation, and relationships between 
5 different multimedia elements disposed in the rendered representation; 

browser processing resources reading the rendered representation and 
the semantic content; 

the browser processing resources restoring a semantic representation 
based on the semantic content; 
1 0 the browser processing resources transmitting: 

an active portion of the rendered representation, the active 
portion of the rendered representation is one of: a portion of the rendered 
representation presently being played, and a portion of the rendered 
representation to be played rapidly after transmitting; and 
15 an active portion of the semantic content corresponding to the 

active portion of the rendered representation; 

client processing resources detecting the active portion of the rendered 
representation and the active portion of the semantic content; and 

the client processing resources playing the active portion of the rendered 
20 representation. 



39. The method for playing multimedia content of claim 38, 
wherein: 

the client includes a set-top box; 
25 the multimedia content includes an image having at least one hyperlink; 

the semantic representation including at least one of: 
a location of at least one hyperlink; 
a size of at least one hyperiink; 
a shape of at least one hyperlink; and 
30 a target index of at least one hyperiink. 
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40. An apparatus for storing multimedia data, comprising: 
means for detecting multimedia content including layout instructions. 

the multimedia content includes at least one multimedia element, the at least 
one multimedia element includes at least one of graphical images, audio, text, 
5 and full motion video; 

means for laying out the multimedia content according to the layout 
instructions to form rendering instructions and a semantic representation of a 
rendered representation of the multimedia content, the semantic representation 
describes at least one of: characteristics of the rendered representation of the 
1 0 multimedia content, and relationships between different multimedia elements 
disposed in the rendered representation of the multimedia content; 

means for rendering the multimedia content according to the rendering 
instructions to produce the rendered representation; 

means for storing the rendered representation; and 
15 means for storing data corresponding to the semantic. 

41. A rendered cache comprising: 

means for storing data corresponding to multimedia content, the data 
including: 

20 a rendered representation of the multimedia content formatted for 

rapid play, the multimedia content includes at least one multimedia element, the 
at least one multimedia element includes at least one of graphical images, audio, 
text, and full motion video; and 

semantic content of the multimedia content, the semantic content 
25 includes data describing at least one of: characteristics of the rendered 
representation, and relationships between different multimedia elements 
disposed in the rendered representation 

indexing means for storing and retrieving: 
the rendered representation; and 
30 the semantic content. 
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FIGURE 5B 
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Fig 6A: Presentation timeline 
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FIGURE 6B 
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FIGURE 6C 
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FIGURE 6D 
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